Echoes - Starting up Ebrow

Starting up Ebrow

Index

Opening a database
Classifications
Creating subsets
Considerations about concurrency
Considerations about multiple instances
Time coverage

Opening a database

Ebrow window, first startup

At first start, the Ebrow window looks like in the figure above, showing the first Database tab. Similarly to what happens with Echoes, Ebrow also creates its own working folder ebrow/in the user's home and - inside it - the settings file ebrow.ini.
From here, the first thing to do is to open a database file produced by Echoes. We can find them into its working folder. At least one database file echoes/default/default.sqlite3 should be available, provided that Echoes has run at least once. So let's open one by pressing the button Open Data Source and selecting it into the dialog window that pops up.

The same, just after opening a database file

In the case shown, I opened from my Windows laptop the GABB's station database, running under Linux in a dedicate headless machine. Its working directory is shared with Samba so I can access its database by mounting a network drive. Anyway, this is Windows/Linux stuff; what matters here is that a database file has been opened, called AUTO_GRAVES.sqlite3.

DB opening details

The above messages are telling us how many days are stored in the database and the total of collected events, 2168 events in three days. This is a raw summary; that means that it's enough a single event happened in a day to count one day of data.
In other words, "one day" means no "one FULL day". The first event (#1) could be happened at any hour of that day (2023-07-06) and the last one (#132273) at any hour of today, before the current hour.

The incomplete events are events that are not covered with all the three states lines (Raise/Peak/Fall); this happens when Echoes hasn't been closed gracefully or due to bugs in the program. You can see, in my station this happened quite often (163 incomplete events) because I test the program there before releasing it to public.
The incomplete events remain in the database until data expiration (by default is 366 days, but it can be modified by the user), in order to keep a continue ID numeration. They won't be present in snapshots and neither in subsets (see below).

Classification

"Performing classifications" and "Classifications loaded" tell us that Ebrow performed a task that was previously in charge of Echoes and now has been moved to Ebrow: the classification of the events.
Looking at each event's information, taken from automatic_data table, and applying the filter settings (under Preferences tab), Ebrow subdivides the events in the following classes:

Overdense
Underdense
Fake (false positive)

Fakes are subdivided in the following subclasses, depending of their guessed cause:

ESD (electro static discharges)
Carrier 1 (continuous signals in capture interval - criteria 1)
Carrier 2 (continuous signals in capture interval - criteria 2)
Saturations (artifacts due to power transmitters nearby)
Too long (lasts too long to be a real meteor event)

These filters are far from perfect, and their parameters need adjustments based on how the Echoes radio is tuned: bandwidth, subsampling, resolution, etc. Therefore it is better not to touch them, possibly if they malfunction it is better to exclude them until a stable radio configuration is reached, and only then try to modify one at a time.

The effectiveness of the new parameters can be verified by resetting the classifications already made (by pressing Reset Classification in Preferences Tab), saving the database (pressing Update Data Source) and reopening it (pressing Reload Data), so that it can perform the classification again.

The ability to reclassify events is one of the benefits of separating data analysis from data collection.
On the other hand, there are also disadvantages; for example, if classifications are not saved frequently - at least every day - the classification process ends up taking up too much time and could generate problems if executed on a running database, with Echoes currently writing on it.

Creating subsets

This functionality is available since 0.1.62. It allows to save database subsets, covering only the interval specified in Time Coverage control. Such subsets contain only complete events and can be later be reopened by Ebrow. Due to incomplete events removal, the events ID numeration can present holes.

Considerations about concurrency

In fact there are two situations in which simultaneous write accesses to the database file are expected.
The first situation is the midnight swap, during which the expired images and dump files are first deleted from the database and then those recorded in the day just passed are loaded. Unlike what happened in previous versions of Echoes, the acquisition is no longer stopped during the swap, therefore all the above described happens while the program continues to record new events.
The second situation is the data access by Ebrow with the possibility of saving the classifications made by it.

The Echoes database uses the Write-Ahead Logging (WAL) option in order to support concurrency, but there is an important rule to keep in mind when opening the running DB files with Ebrow, as the WAL page says:

All processes using a database must be on the same host computer; WAL does not work over a networked filesystem.

This means that if Ebrow and Echoes are running together on the same machine, there shouldn't be any such problems. But close attention is required when accessing a running database from another computer, writings on the DB should be avoided. Maybe nothing happens, you do it even 100 times, the 101st you find yourself with crashed programs and the database file to throw away. I say this from experience.

So my advice is to disable the classifications self saving option from the Preferences tab when you start Ebrow from a computer other than the one running Echoes. And when you quit the program with the Quit key, always press "cancel" on the dialog box which will ask you if you want to save the classifications in the database.

Instead, when Ebrow and Echoes runs on the same computer - for example to automatically send data to the RMOB server - the classifications self saving option is advisable to activate it because in this way the counts sent to RMOB will have already been filtered after discarding false positives.

Considerations about Ebrow's multiple istances

By default, Ebrow can only run in a single instance, that is, there cannot be two Ebrow running simultaneously on the same machine. I decided in this sense because it happened on the GABB station that - due to an error in the database - Ebrow went into exception presenting the relative window requiring the user to press the ok button to exit the program.
This is normally not a problem, but it becomes a problem when Ebrow is launched from cron at fixed intervals in headless mode, as in the case of sending data to RMOB: the memory fills up with crashed Ebrow sessions.

This limit allows you not to start further Ebrow sessions after the first one has crashed, allowing the system to remain operational and reachable from the network. The limit can however be worked around with the --multiple option.

Be careful with Linux, because if Ebrow crashes, the operating system does not automatically free the allocated shared memory, so it will no longer be possible to launch Ebrow until you reboot or specify the --multiple option. This don't happen on Windows systems.

Time coverage

The Time Coverage controls allows to select a portion of the database to work on. By default, when no DB files are opened, the suggested dates are the first of the current month and the current day. Once loaded a DB, the suggested dates become those of the first and the last events present in the database.

When the database covers several months, it becomes convenient to restrict the range, by selecting only the last X days to avoid long waitings while analyzing the data. The new time coverage is stored in the settings file ebrow.ini when closing the program.

The opened database path also is stored into ebrow.ini, so the next time Ebrow is launched, the database will be automatically loaded, selecting the stored interval without needing to press Open Data Source and search for it again.

Beware that the restricted time coverage stored changes when launching the program with --rmob switch, because to generate the RMOB graph, must be considered all the counts of the current month; so in this case the beginning From date will be automatically set to the first of the current month.

The ending date instead is always set to the current date, this because the database normally grows during normal Echoes operation and the most recent events would remain invisible if applying always the stored interval. If the database does not contain today's data - for instance because we're reading a snapshot, the ending date set will be the latest available in the DB.

Note that after opening the database file, other tabs appeared besides Database and Preferences. To analyze the individual events, click the Events tab.