Data Import tutorial (Horiba AquaLog)

This is a tutorial that will demonstrate how to

  • import data from a Horiba AquaLog fluorometer into MATLAB and
  • assemble a dataset.

This tutorial shows focusses on data exported from a Horiba AquaLog. In the AquaLog software, the data export was performed using the HYI export option under the file menu. The advantage of this export is that sample, blank, and absorbance scans are exported per sample and confusion is minimized in this way.

Note: AquaLog instruments come spectrally calibrated from the manufacturer. No emission or excitation correction factors will be applied here. It is your responsibility to ensure proper calibration from time to time.

Also, different instruments may not be correcting for spectral biases themselves and these corrections have to be applied manually!

You can copy & paste the code sections in this tutorial directly into your own MATLAB script. If this document is open in the MATLAB help browser (called with doc), you can right-click on code sections and evaluate them line-by-line (also by pressing F9 on windows)

If you want the code without any comments given in this tutorial, type:

open(which('drEEM_dataImport_AL.m'))

The drEEM toolbox must be installed for this command to work.

The sections of this tutorial

Set up the toolbox

Before we start this tutorial, we need to make sure that the drEEM toolbox is properly installed and the functions are ready to import a example dataset. This set of data is called demofiles.zip and is stored in the drEEM toolbox folders.

If you downloaded the drEEM toolbox to your matlab user folder (the folder returned when you call userpath), simply call

cd userpath

otherwise, say:

cd 'C:\Users\expertuser\somefolder\drEEM\' % Change this!

then, we ensure proper installation of drEEM and availability of the demo files:

dreeminstall
unzip('dreem_demofiles_2.zip',fileparts(which('dreem_demofiles_2.zip')))

Next, we should provide some necessary details and make choices on where the data should be saved. This section can later be modified when different settings are used or the next data import occurs.

datalocation = [fileparts(which('dreem_demofiles_2.zip')),filesep,'demofiles_AL_HYI'];
savelocation = 'C:\Work\folder1\imported\Xin.mat'; % Where should the dataset be saved to?
flusize = 'A2..DR126'; % EEMs limits, open in Excel and note where the EEM begins and ends.
abssize = 'A4..J124';   % Absorbance limits, open in Excel and locate the absorbance column.

Read in fluorescence files

Now that we have the toolbox installed and the information collected, we can begin the data import. First, the samples are imported:

cd(datalocation);                     
[X,Emmat,Exmat,fl,outdata]=readineems(3,'*Waterfall Plot Sample.dat',flusize,[0 1],0,0); 
DS = assembledataset(X,Exmat(1,:),Emmat(:,1),'RU','filelist',fl,[]);
clearvars -except DS datalocation savelocation flusize abssize

Next, the blanks:

[X_b,Emmat_b,Exmat_b,fl_b,outdata_b]=readineems(3,'*Waterfall Plot Blank.dat',flusize,[0 1],0,0);
DSb = assembledataset(X_b,Exmat_b(1,:),Emmat_b(:,1),'RU','filelist',fl_b,[]);

Let’s clean up a bit before we continue

clearvars -except DS DSb datalocation savelocation flusize abssize

Read in absorbance files

In this next step, we read in absorbance scans. They are automatically marked with the filename '* - Abs Spectra Graphs' for this specific instrument.

[S_abs,W_abs,wave_abs,filelist_abs]=readinscans('Abs','dat_1_10',abssize,0,0,'* - Abs Spectra Graphs');

Next, we generate an absorbance dataset

A=[wave_abs;S_abs];          %add wavelengths to Absorbance scans
Abs.A=A;                     %wavelengths are in the first row. They will be removed later.
Abs.filelist=cellstr(filelist_abs);
Abs.wave=wave_abs;

Let’s clean up a bit before we continue

clearvars -except DS DSb Abs datalocation savelocation flusize abssize

Check file alignment

Even though the AquaLog automates file export, it can happen that some files go missing. Since they were separately imported, we must check that all samples match up correctly: eems, blanks and Abs. Otherwise, we will see an error message later on, when all data are combined into one dataset. This section of code intends to give more specific information on which type of file may be missing.

Also: Always confirm that files are matched correctly. Otherwise, a sample may be matched with the wrong blank or an inappropriate absorbance scan!

This is the reason a pause was written into this code.

try
    disp([cellstr(num2str((1:DS.nSample)')) DS.filelist DSb.filelist])
catch
    error('Missmatch: DS.nSample, DS.filelist, DSb.filelist')
end
try 
    disp([DS.filelist DSb.filelist Abs.filelist])
catch
    error('Missmatch: DS.filelist, DSb.filelist, Abs.filelist')
end
try
    disp([cellstr(num2str((1:DS.nSample)')) DS.filelist DSb.filelist Abs.filelist])
catch
    error('Missmatch: DS.nSample, DS.filelist, DSb.filelist, Abs.filelist')
end
warning('Check that the samples are aligned, press enter when ready')
pause;

Resize EEMs depending on absorbance data

If absorbance was measured 240-600 then you can not calculate inner filter effect correction factors for any other wavelengths. These data would have to be deleted.

disp('Abs wavelengths: min and max')
AbsRange=[min(Abs.wave) max(Abs.wave)];
disp(AbsRange)
disp('EEM wavelengths: min and max')
EEMRange=[min([DS.Em' DS.Ex']) max([DS.Em' DS.Ex'])];
disp(EEMRange)

Now, we remove wavelengths below 240 and above 600 from DS and Blanks

Xin=subdataset(DS,[],...
	logical(double(DS.Em<AbsRange(1))+ double(DS.Em>AbsRange(2))),...
	logical(double(DS.Ex<AbsRange(1))+ double(DS.Ex>AbsRange(2))));
B=subdataset(DSb,[],...
	logical(double(DS.Em<AbsRange(1))+ double(DS.Em>AbsRange(2))),...
	logical(double(DS.Ex<AbsRange(1))+ double(DS.Ex>AbsRange(2))));

Correct EEMs

Next, we correct the EEMs for the presence of inner filter effects and normalize EEMs by the Raman peak intensity. In drEEM, this is all done by fdomcorrect.

The data in this tutorial do not contain an excitation scan at Ex = 350 nm, so we will extract the Raman scan in Blanks at 351 nm instead. After extraction, we will add the Raman scans to the blank dataset B.

RamanWavel=351;
Sr=squeeze(B.X(:,:,DSb.Ex==351));
W=[B.Em';Sr];              %2D Matrix of matched (350 nm) Raman scans
B.W=W;


[XcRU Arp IFCmat BcRU XcQS QS_RU]=fdomcorrect(Xin.X,Xin.Ex,Xin.Em,...
[Xin.Em ones(size(Xin.Em))],[Xin.Ex ones(size(Xin.Ex))],... % These are dummy correction factors (ones)
B.W,[351 381 426],Abs.A,B.X,B.W,[],[]);

Consolidate the data

Now, we are ready to assemble the final dataset.

Note: AquaLog instruments are spectrally calibrated. No emission or excitation correction factors were applied here. It is your responsibility to ensure proper calibration from time to time. Also, different instruments may not be correcting for spectral biases!

Xin.X=XcRU;             	%X is in Raman Units 
Xin.RamanArea=Arp;          %Raman Peak Area
Xin.IFE=IFCmat;             %IFE correction factors
Xin.Abs_wave=Abs.A(1,:);    %Abs wavelengths
Xin.Abs=Abs.A(2:end,:);     %Abs data
Xin.Ram_wave=B.Em';      %Water Raman wavelengths
Xin.Emcor='internally corrected by AquaLog'; %corrected by AquaLog not drEEM
Xin.Excor='internally corrected by AquaLog'; %corrected by AquaLog not drEEM
Xin.RamOpt=[350 381 426];   %Raman integration range on Em spectrum
Xin.RamSource='Water Raman scans extracted from blanks'; 

clearvars -except Xin

Finally, we want to make sure that all data are present and the dataset is fully compatible with drEEM.

checkdataset(Xin)

This should return a message like this:

>> checkdataset(Xin) 


Checking the validity of your dataset...
 
     Success: All required dataset variables are present
     Success: The size of your EEM cube is consistent with Ex, Em, and i (incl. nEx, nEm, nSample)
     Success: The fields i, Ex, and Em are correctly stored as column vectors.
     Success: Emission and excitation wavelengths steadily increase.
     Success: EEMs content suggests that Ex and Em information is correct.
     Success: NaN-percentage < 20%
 
Success. The analysis indicates that your dataset is correctly formatted.

Save the data

Last step: Save the data!

save(savelocation,'Xin')