Electromagnetic Simulation Laboratory

Name of the course	Online	Lecture	Exercises	Lab
Electromagnetic Simulation Lab	✓	✗	✗	✓

Self-paced programming laboratory for students with interest in electromagnetic simulation and a background in programming with C/C++ or CUDA. The course is a pure online laboratory without lectures. Presence is possible but not mandatory. Participants need some basic equipment to complete the course successfully, i.e. computer system with Ubuntu Linux and optionally if desired a graphics card that supports CUDA. The course material consists of a programming environment including a library with pre-compiled reference codes that allow students to verify their program results for correctness and run time. Students are provided with function prototypes to implement their program code.

Students work on a given optical system, e.g. grating filter, and are supposed to optimize run times while maintaining correctness. Grades are awarded on behalf of the correctness, complexity of the implemented algorithms and - if possible - on the achieved level of speedup. The environment automatically checks for correctness and algorithm speedup. In case of improvements students are supposed to explain their approach, how and under what assumptions or under what limitations the algorithms achieve the improvement.

Programming environment

Cuda-Version	2D/3D	CPU/GPU	scalar/vectorial	Version	Archive
9.1	✓/✓	✓/✓	✓/✓	3.00	[.tgz]

The programming environment is self-developed and non-commercial. It is allowed to use the code under the CC BY-NC-SA license agreement [w3].

CUDA 10.xx on Ubuntu 20.xx LTS has recently shown compile issues with options '-Ofast' and '-ffinite-math-only'. Please read the documentation below in case of problems.

The tar-archive in the table above provides a C/C++ programming environment for Ubuntu-Linux 16.04, 18.04 and 20.xx with CUDA version 9.x for nvcc and CUDA libraries to perform a self-paced programming laboratory. It allows students to implement 2D or 3D Fourier-based electromagnetic propagation methods for scalar or vector waves on a CPU or GPU. The environment compares the results against the results from reference codes and reports success if errors are below 0.1 per mil per spatial sample.

The tar-archive includes a pre-compiled library with reference codes for scalar and vector BPM and WPM on CPU and GPU. Student codes will be linked against this library and trainees run simulations from pre-defined adaptable configuration files to verify the results. The programming environment reports on run-time and memory usage for CPU and GPU devices. Course participants first implement the basic algorithms and then improve code efficiency by advanced programming techniques, e.g. dynamic programming, multi-threading for a given optical system to simulate. Students select the algorithms, programming method and device as appropriate for their level of expertise. Thereby participants work at their level of complexity to master the course successfully.

How to install the environment?

Download and extract the tar achive.

$ tar xvfz UDPMLAB_Vx.xx.tgz

The host system needs support for fftw and gcc for CPU runs and nvcc for GPU runs. In case of GPU runs, the shared library libcufft7.5.so or libcufft9.1.so has to be available on the host system. LD_LIBRARY_PATH needs to be set properly to pick the libcufft shared library. libcufft is only needed if GPU runs shall be performed!

If libcufft is needed ...

$# this is just a comment
$ dpkg -i libcufft9.1_9.1.85-0ubuntu1_amd64.deb # install package
$ dpkg --contents libcufft9.1_9.1.85-0ubuntu1_amd64.deb # not needed but nice to know
$ dpkg -l # not needed but nice to know

If the cufft library is not available on your system and cannot be installed to the official system directory, get the library from the internet and extract it into a user subdirectory, e.g. the ./lib directory of the tar-archive. Then add the path to the environment variable LD_LIBRARY_PATH.

$# this is just a comment
$ dpkg -x libcufft9.1_9.1.85-0ubuntu1_amd64.deb ./lib # extract package to ./lib
$ export LD_LIBRARY_PATH =../lib:$LD_LIBRARY_PATH

Alternatively but maybe not the best choice due to side effects, redirect the installation to a local working directory.

$ dpkg -i --instdir=YOUR_DIR libcufft9.1_9.1.85-0ubuntu1_amd64.deb # install package to ./lib
$ find YOUR_DIR -name "*cufft*.so" -print # find LIBCUFFT_PATH to shared library
$ export LD_LIBRARY_PATH = LIBCUFFT_PATH:$LD_LIBRARY_PATH

The reference code in the package has been compiled with libcufft7.5 and libcufft9.1. If libcufft cannot be installed at all, user code will compile but errors will occur at runtime for GPU runs, i.e. option '-g'. CPU runs are not be affected. In this case, option '-g' cannot be applied.

If versions of nvcc and libcufft mismatch ...

In some rare cases, versions of nvcc and lubcufft does not match and linker errors occur, e.g. undefined reference to `cudaSetupArgument'. In such cases, versions of nvcc and libcufft need to be aligned.

./bin$ nvcc --version # Cuda compilation tools, release 9.1, V9.1.85
./bin$ ldd ./pm # libcufft.so.9.1 => /usr/lib/x86_64-linux-gnu/libcufft.so.9.1

Versions can be changed by the following procedure ... NVIDIA_CUDA_Toolkit_9_1_Ubuntu_20_04.txt.

Files in the programming environment

The programming environment consists of the following files

./README.md

Quick introduction and FAQ

lib_UDPM_V0.01.a

Pre-compiled static library with reference code and support functions.

Makefile

Makefile to compile the student code against the reference code.

your_pm.hh, your_pm.cc

Function prototypes for student CPU codes. Function prototypes are pre-defined for 2D or 3D and scalar or vector waves and cannot be modified.

your_kernel.hh, your_kernel.cc

Function prototypes for student GPU codes. Function prototypes are pre-defined for 2D or 3D and scalar or vector waves and cannot be modified.

./bin/pm

Executable

What options are supported?

The programming environment supports the following options.

./bin$ ./pm --help

prints the manpage.

How to implement and execute own code?

Write your code into your_pm.cc for CPU code and your_kernel.cu for GPU code. Then remake the executable and run your code

./bin$ cd ..
./$ make
./$ cd bin
./bin$./pm -p -c -t -i wguide.txt

How to invoke the simulator?

To start a simulation, you need to specify a configuration file. Such files are provided with the tar-archive for simple system configurations like for example a waveguide, i.e. wguide.cfg.

./pm -p -i wguide.cfg

and supports the following options To start a simulation on your CPU and compare against the CPU reference code, invoke the simulator as follows

./pm -p -c -i wguide.cfg

For more examples click here.

How to configure a simulation?

Global parameters are set in the global section and some parameters are only applicable to the GUI-version of the tool. The relevant parameters for the batch-version are

[global]
nx = 48
ny = 48
nz = 96
x = 4000
y = 4000
z = 8000
method = vwpm
scale = nm
evanescent = 1
[/global]

Define one or more electric input field distributions by one wave section per distribution. Configuration files are parsed line-by-line. Please do not specify more than one parameter per line.

Gaussian beam	Plane wave	User-defined
[wave] shape = gauss sigmax = 1250 sigmay = 1250 x = 0 y = 0 z = 0 theta = 0 phi = 0 beta = 0 amp = 1 lambda = 850 [/wave]	[wave] shape = plane x = 0 y = 0 z = 0 theta = 0 phi = 0 beta = 0 amp = 1 lambda = 850 [/wave]	[wave] shape = user ... [/wave] User-defined wave-sections and layer-sections are under work and will be available soon.

Layer properties are defined by their real and imaginary parts of the refractive index. Secure that the sum of lengths match the definition in the global section.

Homogeneous medium	Lens	Waveguide	Taper	2D Grating
[layer] type = homog z = 1000 n = 1.0 k = 0 [/layer]	[layer] type = lens x = 0 y = 0 z = 2000 r = -1800 n = 1.6 k = 0 ne = 1.0 ke = 0 dx = 0 dy = 0 dz = 0 [/layer]	[layer] type = waveguide x = 0 y = 0 z = 3000 r = 500 n = 1.6 k = 0 ne = 1.0 ke = 0 dx = 0 dy = 0 dz = 0 [/layer]	[layer] type = taper x = 0 y = 0 z = 11000 rin = 1000 rout = 200 n = 1.6 k = 0 ne = 1.0 ke = 0.0 dx = 0 dy = 0 dz = 0 [/layer]	[layer] type = grating x = 0 y = 0 z = 2000 px = 10000 py = 10000 wx = 5000 wy = 5000 n = 1.6 k = 0.03 ne = 1.0 ke = 0 dx = 0 dy = 0 dz = 0 [/layer]

User-defined layer-sections are under work and will be available soon. Simple configuration files can be found here.

How to analyze the energy flux?

All options in a section [flux] are only supported by Version 3.00 or higher and only for the VWPM, i.e. if 'method' in section [global] is assigned 'vwpm'. Open a flux section in the configuration file with [flux], define the parameters for flux analysis and close the section with [/flux]. The following parameters are supported:

[flux]
do_evanescent_like_propagating=0
do_theta_90 = 1
do_tolerance = 0
do_trim = 0
do_verbose = 0
do_skip_evanescent_modes_from_manual_override = 0
do_eflux_from_h = 0
do_limit_nyquist = 1
do_limit_modes = 0
modes = 20
do_mode_map_report = 0
do_mode_map_summary = 1
do_filter_field = 0
do_filter_system = 1
do_flux_report = 1
do_divd = 0
do_divd_report = 0
do_divd_summary = 1
do_spectrum_report = 0
do_spectrum_summary = 1
[/flux]

do_evanescent_like_propagating

Treats evanescent like propagating modes when set to 1 and applies a model (Josefowski 2007) for evanescent modes when set to 0. This is experimental code to show that evanescent modes cannot be treated like propagating modes.

do_theta_90

Applies a threshold window between 89.9 and 90.1 degrees for the evanescent boundary to avoid numeric instabilities from floating point arithmetics.

do_tolerance

Applies a TOLERANCE of 1E-9 on the input of a propagation step (spectrum) and assigns zero on values smaller than TOLERANCE.

do_trim

Applies a tolerance of 1E-6 on the output of a propagation step and assigns zero on values smaller than TOLERANCE.

do_verbose

Returns information in case of NaN values or in the first case when the energy flux does not hold the conservation law.

do_skip_evanescent_modes_from_manual_override

Skip the processing of evanescent modes by manual override. Works only if do_evanescent_like_propagating is assigned zero.

do_eflux_from_h

Derives the energy flux from the electric and magnetic field if set 1 and only from the electric field if set 0.

do_limit_nyquist

Applies a spatial Anti-Aliasing filter.

do_limit_modes

Manually limits the number of modes. To be used with option 'modes'. 'modes' carries the number of modes to be computed. All discrete angles between (excluding) the -10th and +10th mode will be computed if the number of modes is assigned 20.

do_mode_map_report

Creates a full mode map to indicate propagting, evanescent and skipped mode due to various reasons. Read the description in the report.

do_mode_map_summary

Creates a brief summary of various types of modes and shows the mode structure for every layer under investigation. Read the description in the report.

do_filter_field

Applies a spatial filter on the computed electric field of each iteration. The option is to be used with the options 'filter_type' (0=linear, 2=Gaussian), 'filter_sigma' (floating point value to be used with Gaussian filters), 'filter_nx' (x-size in the filter matrix), 'filter_ny' (y-size of the filter matrix)

do_filter_system

Applies a spatial filter on the refractive index distribution of the system before the simulation is started. The result is stored in N_preRun.csv and N_postRun.csv. The option is to be used with the options 'filter_type' (0=linear, 2=Gaussian), 'filter_sigma' (floating point value to be used with Gaussian filters), 'filter_nx' (x-size in the filter matrix), 'filter_ny' (y-size of the filter matrix)

do_flux_report

Creates a standard flux report with the energy flux per layer and other information.

do_divd

Applies a modified spatial frequency vector derived from the first Maxwell equation to consider lateral field dependencies due to refractive index gradients. This is experimental code and therefore can be switched off by setting this option to '0'

do_divd_summary

Creates a brief summary with infformation on the divergence of the electric displacement field D. This option can be used independent from the option 'do_divd'.

do_divd_report

Creates a full (and large) report on the divergence of the electric displacement field D at every position on the grid. This makes the report very large.

do_spectrum_report

Creates a full report on the spectrum amplitude at every position on the grid. This makes the report very large.

What is the output?

The simulation returns the electric field distribution of vector or scalar fields as a comma-separated list, i.e. .csv-files.

../bin$ N.csv

Refractive index of the electric vector field per node for the configured cross-section (option -x) and layer (option -l).

../bin$ EX.csv

X-component of the electric vector field per node for the configured cross-section (option -x) and layer (option -l).

../bin$ EY.csv

Y-component of the electric vector field per node for the configured cross-section (option -x) and layer (option -l).

../bin$ EZ.csv

Z-component of the electric vector field per node for the configured cross-section (option -x) and layer (option -l).

../bin$ E.csv

Amplitude of the electric scalar field per node for the configured cross-section (option -x) and layer (option -l).

../bin$ F.csv

Energy flux per node for the configured cross-section (option -x) and layer (option -l).

../bin$ FZ.csv

Accumulated energy flux per layer Z.

The outputs can be easily imported into other tools for analysis and coloring. The output can be also imported to the GUI-version of the tool. The GUI-version supports lots of other useful features to setup complex systems and analyze the electromagnetic field distribution, the energy flux and irradiance and more.