Updates HackRF native byte buffers with automatic DC removal.
Creates new half-band channel source to replace the cic-based channel source in the heterodyne channelizer. Resolves issue in RTL native buffer processing. Creates a CACHED thread pool and reduces the number of threads for the SCHEDULED thread pool to 2. Changes each of the channel source implementations to use a dedicated thread (Dispatcher class) instead of the scheduled thread pool, to reduce the amount of buffered/retained samples objects in memory. Lots of other small changes and tweaks.
Removes last of Reusable buffer code. Updates DFTProcessor to use a single thread scheduled executor. Updates Polyphase channelizer to use threads instead of an executor. Removes a number of unused classes.
Commenting out window calibration. Scalar is faster in both implementations.
Adds vector and scalar implementations for windows. Repackages window and window type classes.
WIP adds vector implementations for Airspy non-interleaved buffer iterator and calibration plugins for both interleaved and non-interleaved buffer iterators.
WIP adds vector implementations for Airspy interleaved buffer iterator.
Work on developing scalar and vector implementations of airspy sample conversion and airspy native buffer processing.
Introduces new INativeBuffer framework where tuner raw samples are retained in their native (compressed) format until when they're needed and they are converted on-demand into smaller complex sample buffers to aid garbage collection.
Removes all usage of ReusableComplexBuffers and ReusableChannelREsultsBuffers. Updates polyphase channelizer to be more efficient in output channel processing.
Updates ViterbiDecoder to change Map usage to array for memory efficiency.
Updates channel processors to used scalar implementation for gain and mixer, until the polyphase channel results can be fixed to be a multiple of the SIMD lane width.
Resolves final changes for the conversion from interleaved sample arrays to complexsamples buffers for distributing baseband samples to channels and processing chains.
Updates tuner channel sources and all decoders to receive new complex samples buffers, removing the previous reusable complex buffers as the default baseband samples buffer object.
Still to resolve is the ComplexMixer implementations. Needs validation and a calibration implementation with an update to the complex mixer factory.
Updates the DMR decoder to use optimal (scalar vs vector) implementations of filter and AGC controls.
Updates the AM demodulator for separated I&Q float sample processing.
Code review and testing changes.
updates build script for runtime configuration.
Integrates calibration manager with main sdrtrunk application. Updates NBFMDecoder to use new optimized FIR and Decimation filter options.
Updates real half-band decimation filters to select the optimal (scalar vs vector) implementation.
Adds calibration manager plugins.
Adds calibration manager and plugins to determine the optimal (scalar vs vector) implementation type to use for various classes of dsp components.
Adds vector squelching FM demodulator and cleans up the IIR filter and power squelch to use all floats.
Adds vector implementation of FM demodulator. This introduces a new interface that accepts i/q samples as two separate float arrays, versus an interleaved array, which enables more efficient processing. This new complex sample interface will be used across the codebase eventually.
Adds ComplexOscillator and a SIMD version VectorComplexOscillator. Deprecates the LowPhaseNoiseOscillator because the ComplexOscillator is about 35% faster. The vectorized version is about 43% faster when SIMD lane width is 8x.
Adds new vector complex filters and new real oscillators. Note: the RealOscillator using straight Java math sin() function is 3x slower than the existing legacy implementation. Will replace this with a complex oscillator that only uses the real part.
Updates ComplexFIRFilter to be 50% more efficient.
Adds 23 tap filter implementations. Will not add 63-tap implementations because they are slower than the default vector implementation.
Adds 15-tap customized vector half band filters.
Adds initial custom complex vector half-band decimation filters for 11-tap and 15-tap decimation filters..
Adds vector complex half band filter implementations. Note: performance of the generic 64-bit 2-lane filter is horrible and is not included.
Optimizes complex half-band filter to be 27% more efficient.
Implements vectorized real half-band filter implementations.
Implements a new FIR filter class that processes sample buffers instead of single sample at a time.
Refactored all of the calibratons to be time-duration based. Testing on windows i7 gen 3 CPU showed that it took 1000x longer for one of the calibrations to run, versus my AMD 5800x.
Changed gradle jvm args so that the windows plaf export is always applied.
Updates gradle build script to ensure incubator module is included and to cleanup javafx library management for building runtime images.
Testing tweaks and changes.
Adds logging limiter to dispatcher for buffer overflows
Adds User Preferences editor for performing Vector/SIMD calibrations and new startup dialog prompting the user to perform calibrations.
Updates calibration manager and all calibration implementations to add warm-up and test phases and to reduce the overall time it takes to calibrate a system.
Co-authored-by: Dennis Sheirer <dsheirer@github.com>
* Misc minor performance improvements
* Use more concise loop syntax
* Use more concise Collection sort syntax
* Use StringUtils pad methods
* Optimize some loops and double-map-lookups
* Optimize array instantiation
There are two styles to convert a collection to an array: either
using a pre-sized array (like c.toArray(new String[c.size()])) or
using an empty array (like c.toArray(new String[0]).
In older Java versions using pre-sized array was recommended,
as the reflection call which is necessary to create an array
of proper size was quite slow. However since late updates of
OpenJDK 6 this call was intrinsified, making the performance
of the empty array version the same and sometimes even better,
compared to the pre-sized version. Also passing pre-sized array
is dangerous for a concurrent or synchronized collection as
a data race is possible between the size and toArray call
which may result in extra nulls at the end of the array,
if the collection was concurrently shrunk during the operation.
* Cleanup string concatenation
* Use EnumMap where possible
* Use FastMath instead of Math
* Initialize complex things only once
* Reduce method size to aloow more JIT inlining
* remove map of short->float
* Move field to local variable
* Cleanup Map lookup
Co-authored-by: Rob Ekl <xwq876@zebra.com>
* Updates gradle to 5.1.1
* Updated to run application from intellij using installed Java 11.
Resolves issues with Java 9 deprecations.
* Updates gradle build dependencies and other deprecation issues.
* Updates gradle to produce OS-specific builds.
* Updates the beryx runtime plugin to 1.1.1 to resolve issue with zipping
each of the target distributions.
* Import cleanup
* Resolves issue with Look and Feel error on Max OSX
Resolves issue with building OSX package
Resolves issue with excessive logging about JMBE library.
* Initial work on add support for rotating control channels.
* Creates multiple frequency editor control.
* Updates channel source editor to allow user to select multi-frequency
configuration and add multiple frequencies.
* Adds full support for rotating control channels to the P25 channelizer.
Users define multiple frequencies in the channel source configuration.
The decoder monitors the state of the channel for a CONTROL state. If
the state does not stay in the CONTROL state for a user-specified delay
value, then the decoder rotates to the next frequency in the list. This
behavior continues until the decoder state once again changes to CONTROL.
This is NOT a 'scanner' feature.
* WIP on new P25 message framer.
* WIP on new P25 message framer.
* Adds new 1/2 and 3/4 rate trellis decoders. Still trying to chase
down why there is a sync pattern mismatch in the NID buffer queue.
* WIP - new p25 message framer works correctly now. PDU's are dectected
and decoded using the 1/2 and 3/4 rate trellis decoders. Packet
Sequences are assembled. Need to add a new packet sequence message
factory that can detect Motorola Auto Registration Service (ARS)
formatted UDP/ID packets and SNDCP command packets.
* WIP - removes nullify on channel results queue.
* WIP - updates message framer for TSBK 1/2/3 processing.
* WIP removing P25 sync pattern and nid bits from all message parsers.
* WIP - merges identifier framework branch changes into this branch.
* WIP on updating link control words to new format and identifier framework.
* WIP - implements rest of link control words for extended formats
* WIP - initial work on ISP/OSP TSBK message restructuring.
* WIP - adds more TSBK ISP and OSP messages
* WIP - finished updating most of the TSBK ISP and OSP messages. Converted
all messages to no longer include the NID (nac + dataunitID) so that
each message can simply hold the parsed values. Removed AliasList as
an argument to each message and updated all messges to use the new
IIdentifer and IAPCO25Channel interfaces.
Removed legacy TDU and LDU link control messages and replaced with a
single LinkControlWord class hierarchy.
P25 decoder state is not yet updated for the new message structure.
PDU message parsing is not fully implemented.
* WIP - adds some Multi-Block trunking control messages with Alternate
and Unconfirmed formats.
* WIP - Fully implements AMBTC and UMBTC PDU message parsers.
Some package restructuring.
* WIP - Resolves Link control parsing in LDU1 and 2 messages.
* WIP - finishes updating TSBK and Link Control messages. Adds support
for moto link control messages for patch groups.
* WIP - finishes updating TSBK and Link Control messages. Adds support
for moto link control messages for patch groups. Adds support for moto
ARS service, IPV4 decoding, and UDP/IP packets.
* WIP - finishes SNDCP packet data with header parsing. Compressed IP or
UDP headers are not yet supported, but are logged as such.
* WIP - converts all talkgroup identifiers to integers and updates
alias list to create a per-protocol map of talkgroup id's to alias.
Inserts a legacy talkgroup (ltr, p25, passport, mpt1327, mdc, etc.)
conversion utility to convert existing legacy (string) talkgroups to
new integer talkgroup format.
Creates an audio packet manager to insert metadata into audio packets
before streaming packets for playback, recording, and/or streaming.
Creates identifier collection to manage current identifiers in each of
the channel and decoder states.
Creates simplified channel metadata to maintain the now playing display.
Moves channel attributes (alias list, system, site, name) as configuration
identifiers in the channel state.
Many other refactoring and updates.
* WIP - numerous updates to push use of identifier collection throughout
the system. Updates the channel metadata and audio packet processing
logic. Implements a large portion of the P25 decode state processing.
Updates traffic channel management. Adds user preferences for timestamp
and identifier formatting, however doesn't add a user interface yet.
* WIP - initial implementation of user preferences editor window.
* WIP - updates preferences editor window to close and dispose window but
retain javafx event thread running.
* WIP - fully updates p25 decoder state and traffic channel processing to
use the new identifier framework.
Cleans up channel event processing and removes channel model as the sole
broadcaster of channel events. Channel processing manager now listens
for all start/stop events and requestors dispatch event directly to the
channel processing manager.
Resolves issue in airspy that was causing user interface to lock-up. The
issue appears to occur when the dongle is plugged into a usb port that
can't fully support the data rate. sdrtrunk now recognizes these errors
and warns the user that they should use another usb port.
* WIP - updates alias action manager and adds new talkgroup range alias identifier.
Updates playlist to incorporate a version number so that the playlist
updater only runs when needed, to update a playlist to the correct
version format.
Updates user preferences to use the google event bus to broadcast
preference update notifications to remove the need for listeners to
register directly on the user preferences instance.
* WIP - updates alias id editors to flag invalid values and moves legacy
alias ids to legacy package.
Updates event logging configuration to add traffic channel decode events
and decoded message logging. Removes binary message logging since that
is now replaced with demodulated bitstream recording.
* Removes system properties from user preferences. Preferences are stored
in the OS-dependent settings store (windows - registry)
* Updates MapService to use new PlottableDecodeEvent instead of monitoring
all of the decoded messages.
Updates Fleetsync decoder and related messages to identifier framework.
* Updates LTR-Net, Fleetsync and LJ1200 decoders to new identifier
framework and adds support for user preferences for formatting of
LTR-Net and fleetsync talkgroup identifiers.
Breaks LTR-Net messages into various message types and uses a new
message factory.
Removes alias list from message processing for ltr-net, fleetsync and
lj1200.
* Updates LTR-Standard messages to Identifier framework.
* Removes Plottable and IPlottable from codebase.
* wip
* WIP on MPT-1327 messages.
* Updates MPT1327 decoder to identifier framework and new traffic channel
manager. This decoder still needs to be updated with a new message
parser factory and a sync detector like what is now used for P25.
* Updates MDC1200 decoder and message to identifier framework.
* Updates Tait1200 and Passport decoders to identifier framework.
* Merges changes from master branch.
* Removes legacy alias support classes. Updates passport message processor
to only pass valid messages. Adds support to playlist updater for
converting passport mobile ids to become passport talkgroup values.
* Updates P25 decoder state and channel state to better track allocated
traffic channels. Resolves issue in timestamp manager for trackig
message timestamps. Resolves issue in airspy tuner that was causing
the tuner to stop sending samples to the application.
* Update P25 Decoder state to process SNDCP management messages and
PDU messages.
Updates P25 activity summary to display network and site details.
Resolves issue with overlay panel that prevented enabled channels from
rendering correctly when zoomed in.
* Creates a new FilePreferences as part of the user preferences to begin
the transition of SystemProperties over to user preferences and consolidate
file preferences in one location.
Updates playlist manager to use the new file preferences and to use
default.xml as the new default playlist name, following conversion of
a playlist_v2.xml format to the new format.
In the near future, the user will be able to edit file locations and to
create/use other playlists.
Resolved an issue with all decoder states where not all identifiers were
being included in decode events.
* Fix for issue #441.
Method doc reads that the method is thread-safe, but there exist no locking to ensure thread safety on call into recycle method. Modifying the decrementUserCount method to block other callers to ensure thread-safe access.
This appears to be a hot area of code and there may be a performance hit by doing this, please double check my work here.
* Moving synchronized to recycle method as it's accessed by two different methods that could potentially cause an unsafe access race condition.
* Revert - synchronizing recycle does not protect against multiple recycle calls on the same object.
Updated audio packet wave recorder to correctly dequeue audio packets
to ensure that the overflowable buffer queue accounting process was
working correctly.
Updated overflowable buffer queue's poll() method to correctly decrement
the queue size counter.
Resolved issue where the mutable metadata was being prematurely reset
and causing the end audio packet to be mis-identified. This resulted
in the recording manager not stopping the recording on command with the
timeout mechanism kicking in after a brief delay.
channel manager so that channel source requests can have a start
timestamp specified and the channel can be preloaded with samples from
the delay buffer. If no timestamp is specified, current time is assumed.
This feature requires corresponding updates to any decoder that wishes
to implement the timestamped sample preloading functionality.
buffers. None of the decoders currently support this yet, but the P25
will be updated to provide bitstream output for recording.
Renames the ReusableBuffer class to ReusableFloatBuffer for
consistency.
Refactors a MixerReader instance from the RealMixerSource so that it
can be reused for both real and complex data outputs.
Updates RealMixerSource to use refactored MixerReader and use a default
sample rate of 8 kHz.
Updates the ComplexMixer to use the refactored MixerReader.
Updates the FCD 1/2 tuners for compatibility with the new ComplexMixer.
Removes several unused classes.
filter structure. Updated the channel results array buffer to produce
reusable buffers.
This has dropped the overall memory usage considerably when profiling.
Resolved issue in the processing chain where polyphase channel sources
were not being stopped and were stacking up in memory.
was not computationally feasible with my 3rd Gen i7 processor running
and airspy at 10 MHz. Restructured the polyphase channelizer to use
vector iteration structures that the JDK 9 JIT compiler can translate
into SIMD instructions. Separated the IFFT processing from the filter
process so that they can be executed on separate threads and this allows
the processing to occur real time.
P25 C4FM decoding at 25 kHz channel sample rate using an airspy
10MHz sample stream seems to be working nicely with ~25% CPU usage and
~300 to 500 MB memory usage. Incremental (parallel) channel decoding
should have a minimal increase in CPU loading since all of the 12.5 kHz
channels are available in the polyphase channelizer and the only
additional workload is the decoding of the 25 kHz sample stream.
There is still an issue with stopping channels and the channelizer keeps
processing, and occasionally when starting traffic channels something
deadlocks and causes out of memory.
adds Feed forward gain support to the C4FM DQPSK demodulator which seems
to improve output from the interpolating sample buffer at each symbol
period. Updates the symbol timing gain for C4FM to decrease the gain a
bit since the feed forward gain is providing a better interpolated symbol.
updates the LSM decoder to follow the same design pattern. Adds LSM
decoder support to the demodulator viewer application. The current
symbol timing gain for LSM is set too high -- is using the C4FM gain
and causes too much fluctuation in the timing error.
error calculator. After increasing the gain for symbol timing, this
new symbol evaluator class seems to track the signal very accurately
despite lots of noise and ISI.
based on the P25 LSM decoder using a costas loop and a gardner detector.
Updates the Demodulator Viewer FX application to add in a symbol chart
and a phase error chart. The decoder won't lock on yet ... still a wip.
when the tuner controller indicates that there is a center DC spike
blocked region or when the tuner has clipped outer spectral regions
resulting in less than 100% usable bandwidth.
produces a significant reduction in memory usage, especially for tuners
like the airspy and hackrf that produce 10/20 MSPS sample rates. This
is observed to be around 66% reduction in peak memory usage and provides
a more stable memory utilization across time reducing stress on the
garbage collector.
Use of these reusable memory buffers will be pushed further down into
the processing chain modules to ensure we keep new object creation
to the lowest levels possible.
calculator must be updated immediately any time the tuner's center
frequency changes so that a channel can be allocated immediately. So,
I changed source event handling to two parts where the channel calculator
is immediately updated and the channelizer configuration is deferred so
that it can happen on the buffer processor execution thread and we can
avoid any sync() locking on the channelizer or polyphase channel
sources.
messages and events can be accurately timestamped, and more importantly,
allocated traffic channels can request to be pre-loaded with buffered
samples to ensure that traffic channels don't miss any samples that
are currently missed due processing latency inherent in current buffer
sizing. This should reduce the clipping of starting messages or voice
on the traffic channels.
Restructures processing path for tuner frequency and sample rate change
events. The tuner now embeds these events in the sequence of buffers
that are generated immediately after the event occurs since those are
the buffers that will be impacted by the change. Previously, the change
event would be broadcast throughout the system and would impact any
buffered/cached buffers. This also reduces the number of source event
change listeners that have to be managed. Note: this change is not yet
implemented in all tuners ... only in the TestTuner implementation.
Updates the polyphase channelizer to now buffer channel results and then
dispatch the buffer to the channel output processors to improve efficiency
of using float arrays versus single value complex float processing.
