emLib -Crc

Segger emLib Compress

emLib
emCompress is a compression system that is able to reduce the storage requirements of data that must be embedded into an application. Compress on the host in advance, decompress in the target on demand.

A compressed version of the data is stored in the flash memory of the target system. In the target, a small, fast decompressor can decompress the data on-the-fly, when ever it is required. The decompressor can decompress into RAM or send the output to an application defined function. It can be used with little RAM on the target side for decompression; the amount of RAM available to the decompressor is taken into account during compression and an appropriate compression algorithm is automatically selecte

Highly efficient compression
Small decompressor ROM footprint
Fixed decompressor RAM use, chosen when compressing
Wide range of codecs to choose from
Automatic selection of best codec for each file
Easy-to-understand and simple-to-use
Group mode compression boosts small file compression ratios
Simple to configure and integrate
Royalty free

Why should I use emCompress?

Simple: emCompress is able to significantly reduce the amount of storage required for data in embedded systems, by compressing it during compile time. This can save flash, cost, data transmission times or even make it possible to fit the application into the available space.

Typical uses of emCompress are:
Configuration bitstreams to program FPGA and CPLD devices.
Permanent files for embedded web server static content.
Upgrading firmware using a compressed image.
Storing user interface messages for multiple languages.

Software only grows in one direction

With increasing complexity of today's devices, customers expect firmware updates over the life of a device. It's important to be able to replace both firmware images and FPGA configuration bitstreams in the field. Typically, firmware upgrades and other static content only grow in size over the lifetime of a device: features are added to software, not taken away, it's just what happens. Using quickly-written ad-hoc utilities to convert content directly to compilable code (possibly with some form of simple compression) is usual, but it's towards the end of a project when you realise that the static content just won't fit into your device with the baggage of new firmware features.

emCompress offers a professional solution!

This is where emCompress can help. emCompress will compress your data on a workstation so that it takes much less space on the target device. The decompressors are tiny in ROM, but the benefits of compression means you reclaim more space in your device for the features you're trying to add.
Because emCompress decompressors can be tailored for RAM use, you can decompress static content early in your application and not devote RAM to decompression buffers. For instance, configuring an FPGA is one of the first things that an application will do, decompressing a bitstream and sending it to the FPGA. In this case, a small decompression buffer can be held on the stack even though the compressed bitstream is hundreds of kilobytes in size: the temporary buffer is a known size that is configured at compression time, and that RAM is free to reuse the moment any decompression completes.
Or course, emCompress is not limited to these applications, it effective at reducing any type of stored content.

Group mode offers even better compression!

emCompress can boost compression ratios using group mode when compressing many small files. In group mode, the uncompressed source files are concatenated and compressed as a whole, increasing the opportunities for the compressor to find ways to compress the combined data. This proves especially effective when compressing small, static HTML files for embedded web servers. Decompression for group-mode compression is identical in operation to single-file compression, giving you the flexibility to choose how to compress and decompress your data.

Why choose emCompress?

In contrast to many compression utilities, emCompress does not seek to provide the ultimate in compression ratios and decompression speeds. Rather, emCompress focuses on the memory available to run decompression on the target and choose the best compressor that will work to meet those constraints. It's easy to use, it's simple to integrate, and as with many other SEGGER software products, SEGGER have the confidence to build it into our own hardware products which are in daily use by hundreds of thousands of satisfied users worldwide.

What's in emCompress?

emCompress ships with the compression application to run on Windows, the source code for all decompressors, and a comprehensive user manual.
emCompress is written in well-commented ANSI C and is both compiler and target independent. In contrast to most compression libraries, emCompress is not covered by an open-source or required-attribution license and can be integrated in any commercial or proprietary product without the obligation to disclose the combined source.
Because the decompressors are provided as source code, emCompress decompressors can be inspected by auditors. The compression utility's source is available as an additional module.

Typical example — Configuring an FPGA

In products, where an FPGA is used and programed via the microcontroller, the configuration bitstream has to be stored in the flash memory. Depending on the complexity of the FPGA configuration the bistream can be a few hundred kilobytes, which can be too much to fit into small microcontrollers with less internal flash.

A few hundred kilobytes can be too much for small microcontrollers with less internal flash, but for cost reasons we do not want to use a larger controller or add external memory to your product. The solution is to reduce the storage requirements of the bitstream.

Fitting Data Into Small Flash

The size of our uncompressed configuration bitstream is 360 kByte. Our FPGA supports a compressed configuration bitstream, resulting in 112 kByte of data. We already saved some memory here, but it is still to large for our small flash.

Now emCompress can help us. We compress the 112 kByte bitstream with emCompress and include it into our application.

Compressing and decompressing data requires a workspace in RAM. With emCompress we can select the workspace which is available for decompression while compressing the file.

A larger workspace might result in higher compression rates, but in fact requires more RAM. We will test emCompress with a small workspace of 2 kByte, and an unlimited workspace which will show the largest workspace size.

The best compression we get for our bitstream is 23 kByte, requiring a workspace of 35 kByte to decompress it. With a limited workspace of 2 kByte we get a compressed output of 30.5 kByte, 27.3% of the original file. Since our RAM is limited, too, and 30.5 kByte will fit into our flash, we choose the 2 kByte workspace.

Decompression RAM	Uncompressed	2 kB	48 kB
Data	Uncompressed	emCompressed
Data size	112 kByte 100%	30.5 kByte 27.3%	23 kByte 20.6%
Compression codec		LZJU90	DEFLATE
Decompression speed¹		3 MByte/sec	2 MByte/sec

1: Running on Cortex-M4 @ 200 MHz

Decompressing and Processing Data

emCompress features two decompression functions.
The first one is decompression into memory. The complete data is decompressed and stored in a user-provided memory buffer. Although the buffer can be temporary, this requires to have enough free memory for the complete uncompressed data and the workspace. Decompression into memory can for example be useful for dynamic firmware images.

The second function is decompression in streamed mode. emCompress will use a small temporary buffer whose size is set by the user when compressing. Once a fragment of data is decompressed and the buffer is filled, the user-provided output function is called and the next buffer filled again with the next fragment. Sreamed decompression is particularly effective for programming FPGAs or serving web content.

To process our bitstream and send it to the FPGA, we do not have to do this in one go. So we can choose decompression in streamed mode and send the data fragment by fragment in the output function. We do not have to have free RAM for the complete decompressed bitstream (112 kByte) and can simply pass our output function FPGA_SendData() to the decompressor.

To see how this is done and how easy emCompress is to use, look at the example application below.
Decompression is not as fast as copying data or calling an output function directly, but for large data, which is processed once in a while, like a one-time setup of the FPGA while booting, the advantage of less space is more important than the time taken to decompress. The emCompress codecs are highly optimized to achieve fast speeds. Like the compression ratio, the decompression speed is dependent on various factors, like the data to be decompressed, the chosen codec and the temporary buffer size, and can vary. We decompressed our bitstream and calculated a checksum in the output function (without sending data to the FPGA) in 36 ms.

Performance and Memory Footprint

ROM use
The amount of ROM that emCompress uses for decompression varies with the codec selected. The table below measures the total ROM required for a single decoder, including all supporting functions and excluding integrity checks.

Some of the codecs share common code. Hence, if your compressed bitstreams only use DEFLATE, the amount of ROM required is easily read off from the table. If you are using more than one codec in your application, the amount of ROM is less than the sum of the used codecs.

Adding a table-driven CRC32 integrity check adds about 1.1 KB of code to the total emCompress ROM requirement.

The following table lists the ROM requirements per codec.

Codec	ROM
STORE	0.5 KByte
RLE-PAR	0.8 KByte
HUFF	1.3 KByte
LZW	1.0 KByte
LZSS	1.2 KByte
LZJU90	1.3 KByte
DEFLATE	2.1 KByte

RAM use

The amount of RAM that emCompress uses is under complete control as it is specified at compression time. Typically, 2KB of temporary RAM for decompression already yield good compression ratios, but even without temporary RAM, good compression ratios can be achieved in many cases. No static RAM is required, stack usage is well below 512 bytes.

Performance

The compression ratio and decompression speed are dependent on various factors and may vary in each case.
The main factors are:
The data to be compressed
The data size and redundancy
The compression codec
The decompression workspace size
emCompress can achieve best-possible compression ratios of 90% and above as well as decompression speeds of up to 5 MByte/sec.

FAQ

Q:	emCompress is great! Where can I find the compressor sources?
A:	emCompress is designed for fast and effective decompression at runtime and the companion emCompress application is designed to run on a workstation or PC where memory is plentiful. Because the emCompress application is all that is required for preparing data to be decompressed, we do not ship the source code of the compressors built into the emCompress application.

Q:	I still need compression in my application, though...
A:	Should you wish to take advantage of on-device compression in your application, please contact us to discuss your requirements and how we might be able to help.

Q:	Why are there so many compressors?
A:	emCompress ships with compression algorithms that can be parameterized to tune decompressor memory. As each decompressor has different memory requirements and compression capabilities, emCompress covers a wide range of use from excellent compression using moderate amounts of RAM to good compression using tiny amounts of RAM. No one compression algorithm will be applicable across the wide range of inputs and workspace requirements of target applications.

Q:	What's the purpose of the STORE compressor?
A:	The STORE compressor ensures that incompressible data does not expand, as it generally does using other lossless compressors. It allows incompressible data to be handled in the same manner as compressed data, freeing the client from treating uncompressed data differently from compressed data.