Releases: idaholab/Malcolm
Malcolm v24.12.0
Malcolm v24.12.0 contains several improvements to the Malcolm configuration script, the Malcolm user interface, and the Malcolm API, as well as component version updates and bug fixes. This release also corresponds with the release of malcolm-test
(cisagov#486), a Malcolm system testing framework.
- Features and enhancements
- Creation of a Malcolm systems testing framework (cisagov#486)
- Added a number of Zeek packages to detect various CVEs
- Improvements to the Indices, Ready, and Document Ingest Statistics APIs
- Use new arkime tag-hiding feature to hide
netbox
tag from UI (cisagov#495) - Provide configuration script options for pulling from threat intel feeds (cisagov#532)
- Prompt during configuration whether to enable capture statistics (cisagov#504)
- Add additional EVTX fields to index template (cisagov#525) and minor improvements to normalization
- Add simple readiness indicator to upload page (cisagov#528)
- Add option to upload page to disable NetBox enrichment for the currently-uploaded batch of PCAPs
- Expose more of the Logstash API passthrough to the Malcolm API
- Component version updates
- Arkime to v5.5.1
- capa to v8.0.1
- elasticearch-dsl Python library to v8.17.0
- elasticsearch Python library to v8.17.0
- Fluent Bit to v3.2.2
- NetBox to v4.1.8 (major update from the v4.0.x series, see cisagov#496)
- opensearch-py Python library to v2.8.0
- yq to v4.44.6
- Zeek to v7.0.5 (security and bugfix release)
- Bug fixes
- Zeek DNS records don't open correctly in Arkime sessions (cisagov#509)
- Fixes to some Zeek
dns.log
parsing conflicts between ECS's DNS fields and what the Arkime schema is expecting
- Fixes to some Zeek
- Mandiant threat intel source doesn't get split correctly when using JSON zeek log format (cisagov#494)
- Set
indices.query.bool.max_clause_count
to 8192 to reflect maximum number of fields - Increase Java stack size (
-Xss
) for Logstash from1536k
to2048k
- Minor fixes for parsing Zeek
intel.log
(some fields not named correctly with Zeek JSON-formatted logs) - Fixed setting the
Signature
event severity tags
- Zeek DNS records don't open correctly in Arkime sessions (cisagov#509)
- Code and project maintenance
- Replaced hard-coded Malcolm version number in documentation markdown files with variable-based replacer populated during generation
- Documentation and screenshot updates
Official ISO installer images for Malcolm and Hedgehog Linux can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split into 2GB chunks and can be reassembled with scripts provided for both Bash (release_cleaver.sh
) and PowerShell (release_cleaver.ps1
). See Downloading Malcolm - Installer ISOs for instructions.
Malcolm v24.11.0
Malcolm v24.11.0 contains a new threat intelligence feed integration, a few new API calls, other minor improvements, bug fixes, and component version updates.
- Features and enhancements
- Added
dashboard-export
to the list of Malcolm APIs (cisagov#401) - Added
ingest-stats
to the list of Malcolm APIs (cisagov#488) - Added support for pulling from the Mandiant Threat Intelligence service to feed the Zeek intelligence framework as used by Malcolm's and Hedgehog Linux's Zeek processes. The integration uses the google/mandiant-ti-client library for Python. (cisagov#358)
- Improved normalization of Zeek's
intel.log
to the ECS's threat fields - Improved the Zeek Intel dashboard
- Improved the health/liveness probe for the Logstash container
- Changed behavior of Malcolm's non-live Zeek container (responsible for processing uploaded PCAPs) so that it becomes available to process data even before an intelligence feed pull is finished
- Implemented paging for extracted files download dialog (cisagov#361)
- Implemented support for sending Zeek logs to Kafka using the SeisoLLC/zeek-kafka plugin (cisagov#357)
- Added the NetBox HealthCheck plugin as a default NetBox plugin
- Updated the Malcolm services readiness status API to use the new LogStash health report API and the NetBox HealthCheck plugin as the basis for reporting the state of LogStash and NetBox, respectively.
- Added parsing for the new OPCUA-Binary write subscription service log
- Added
- Component version updates
- Arkime to v5.5.0
- Beats to v8.16.0
- elasticsearch Python library to v8.16.0
- elasticsearch-dsl Python library to v8.16.0
- evtx to v0.8.4
- LogStash to v8.16.0
- OpenSearch and OpenSearch Dashboard to v2.18.0
- watchdog Python library to v6.0.0
- werkzeug Python library to v3.0.6 to address CVE-2024-49767 and CVE-2024-49766
- Bug fixes
- Fixed an issue with the
./scripts/configure
script not prompting to regenerate the internal NetBox passwords when it should have - Fixed errors when running
malcolm_appliance_packager.sh
on macOS (cisagov#492, thanks @robrui)
- Fixed an issue with the
- Configuration changes (in environment variables in
./config/
) for Malcolm and incontrol_vars.conf
for Hedgehog Linux- Malcolm
ZEEK_KAFKA_ENABLED
,ZEEK_KAFKA_BROKERS
, andZEEK_KAFKA_TOPIC
have been added to./config/zeek.env
, which can be used to enable Zeek's sending of its logs to Kafka (cisagov#357)ZEEK_DISABLE_DETECT_ROUTERS
(default value:true
) has been added to./config/zeek.env
which controls an experimental Zeek script for detecting the presence of routers (logging them toknown_routers.log
) in a network based on packet TTL; it is recommended to leave this set totrue
as this script is not yet ready for general production useZEEK_INTEL_REFRESH_ON_STARTUP
has been renamed fromZEEK_INTEL_REFRESH_ON_ENTRYPOINT
in./config/zeek.env
to more accurately reflect the purpose of the variable
- Hedgehog Linux
ZEEK_KAFKA_ENABLED
,ZEEK_KAFKA_BROKERS
, andZEEK_KAFKA_TOPIC
have been added tocontrol_vars.conf
for the same purpose as described above
ZEEK_DISABLE_DETECT_ROUTERS
(default value:true
) has been added tocontrol_vars.conf
for the same purpose as described aboveZEEK_INTEL_REFRESH_ON_STARTUP
has been renamed fromZEEK_INTEL_REFRESH_ON_ENTRYPOINT
incontrol_vars.conf
to more accurately reflect the purpose of the variable
- Malcolm
- Code and project maintenance
- All open issues and the project board have been migrated from the Idaho National Lab fork to the upstream CISA fork. The repos will continue to be kept in sync going forward. (cisagov#350)
Official ISO installer images for Malcolm and Hedgehog Linux can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split into 2GB chunks and can be reassembled with scripts provided for both Bash (release_cleaver.sh
) and PowerShell (release_cleaver.ps1
). See Downloading Malcolm - Installer ISOs for instructions.
Malcolm v24.10.1
Malcolm v24.10.1 contains some minor improvements, a few component version updates, a fix for a regression bug, and a fair amount of code cleanup.
- Features and enhancements
- Update AWS AMI build scripts and demo setup scripts to use Amazon Linux 2023 instead of Amazon Linux 2 (#591)
- Add support for
websocket.log
(#593) - Add a "readiness" API that can be used to determine if various Malcolm services are ready (#598)
- Component version updates
- Bug fixes
- Fixed OpenSearch anomaly detection default detectors not being created (regression, #596)
- Configuration changes (in environment variables in
./config/
) for Malcolm and incontrol_vars.conf
for Hedgehog Linux- Malcolm
ZEEK_JA4SSH_PACKET_COUNT
(with a default of200
) has been added to./config/zeek.env
, which can be used to set logging interval number of packets forja4ssh.log
(#508)
- Hedgehog Linux
ZEEK_JA4SSH_PACKET_COUNT
has been added tocontrol_vars.conf
for the same purpose as described above
- Malcolm
- Code and project maintenance
- Examine distro hardening, fix and update documentation as needed for Malcolm and Hedgehog Linux ISO-installed environments (#328)
- Refactoring and code cleanup in the Logstash Zeek pipeline (#592)
- Logstash container initialization code now automatically ensures that the Zeek TSV log parsing filters (
dissect
andsplit
filters) in these files are looking for TAB characters (i.e., automatically replace spaces with tabs in these filter files in case the author forgot to do so) (#592) - Did some code cleanup in the
./shared/bin
directory, mostly moving things that were specific to either the Malcolm or Hedgehog Installer ISO environments out of shared and into their respective locations for the ISO installer build. - When doing the
aquasecurity/trivy-action
action, useTRIVY_DB_REPOSITORY: ghcr.io/aquasecurity/trivy-db,public.ecr.aws/aquasecurity/trivy-db
to try to fall back to an alternative official location for the vulnerability database if the first one fails. Also, pin this action to thev0.28.0
release rather than setting it tomaster
. - As it's used pretty ubiquitously in shared scripts by many of the Malcolm containers, the
jq
utility is now installed across the board during the container image build. - Added a script to gather GitHub API metrics for Malcolm downloads (#594)
Official ISO installer images for Malcolm and Hedgehog Linux can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split into 2GB chunks and can be reassembled with scripts provided for both Bash (release_cleaver.sh
) and PowerShell (release_cleaver.ps1
). See Downloading Malcolm - Installer ISOs for instructions.
Malcolm v24.10.0
Malcolm v24.10.0 contains fixes for a few regression bugs, minor improvements, and a few component updates.
- Features and enhancements
- Enable Zeek's parsing of HTTP server and client header names as
zeek.http.client_header_names
andzeek.http.server_header_names
- Bumped maximum field limit in OpenSearch templates from 5000 to 6000
- Some documentation improvements
- Build improvement: fall back to alternative Zeek .deb download URL (#585)
- Build improvement: limit threads for spicy build processes during Zeek package installation (#571)
- Enable Zeek's parsing of HTTP server and client header names as
- Component version updates
- Bug fixes
Official ISO installer images for Malcolm and Hedgehog Linux can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split into 2GB chunks and can be reassembled with scripts provided for both Bash (release_cleaver.sh
) and PowerShell (release_cleaver.ps1
). See Downloading Malcolm - Installer ISOs for instructions.
Malcolm v24.09.0
Malcolm v24.09.0 contains new features and enhancements, component version updates, and bug fixes.
- Features and enhancements
- When building Docker images and the Hedgehog Linux ISO, allow specifying alternate download URL for MaxMind GeoIP database files (#565)
- Allow total index size-based pruning for
opensearch-remote
andelasticsearch-remote
database modes (#446) - Allow splitting out indexes by other field values (#450)
- Allow users to use the Arkime Lua plugin without having to create new bind volume mounts manually (#533)
- Automatically create empty document on startup to avoid "no data" message spamming by Dashboards (#527 and #567)
- Improvements to documentation and
install.py
for Linux performance tweaks (#495) - Include netbox-topology-views plugin by default (#553)
- Integrate HART-IP parser (#561)
- Add option to go backwards in Malcolm's dialog-based
install.py
installation and configuration script (#487) - Added Podman support (#407)
- Update EtherNet/IP and CIP to account for new packet correlation ID (#558)
- Update Network Traffic Analysis with Malcolm slides
- Component version updates
- watchdog Python package to v5.0.x (#550)
- supercronic to v0.2.32
- osd_transform_vis to v2.16.0
- Fluent Bit to v3.1.8
- OpenSearch and OpenSearch Dashboards to v2.17.0
- YARA to v4.5.2
- Zeek to v7.0.1
- Spicy to v1.11.1
- elasticsearch-dsl Python package to v8.15.3
- elasticsearch Python package to v8.15.1
- Beats to v8.15.1
- Logstath to 8.15.1
- flask-cors Python package to v5.0.0 to address CVE-2024-6221
- Bug fixes
- Filtering on hunt ID in Arkime not working (#554)
- Hedgehog with OOB/VPN connection sets
ARKIME_NODE_HOST
incorrectly (#560 and #559, thanks @divinehawk) - Offline
suricata
Docker container does not initializesuricata.yml
config file (#564)
- Configuration changes (in environment variables in
./config/
) for Malcolm and incontrol_vars.conf
for Hedgehog Linux- Malcolm
- The
MALCOLM_NETWORK_INDEX_SUFFIX
andMALCOLM_OTHER_INDEX_SUFFIX
variables in./config/opensearch.env
now also support expanding dot-delimited field names in{{ }}
(e.g.,{{event.provider}}%{%y%m%d}
). MALCOLM_CONTAINER_RUNTIME
has been added to./config/process.env
to indicatedocker
,podman
, orkubernetes
. This value only currently used in the install, configuration, and control scripts, not inside the containers themselves.ZEEK_DISABLE_ICS_HART_IP
has been added to./config/zeek.env
and can be set totrue
to disable the new HART-IP protocol parser.
- The
- Hedgehog Linux
ZEEK_DISABLE_ICS_HART_IP
has been added tocontrol_vars.conf
and can be set totrue
to disable the new HART-IP protocol parser.
- Malcolm
Official ISO installer images for Malcolm and Hedgehog Linux can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split into 2GB chunks and can be reassembled with scripts provided for both Bash (release_cleaver.sh
) and PowerShell (release_cleaver.ps1
). See Downloading Malcolm - Installer ISOs for instructions.
Malcolm v24.08.0
Malcolm v24.08.0 contains minor improvements, some component version updates, and bug fixes.
- Features and enhancements
- in ISO installer, prompt to format other drives for artifact storage rather than just doing it automatically (#529)
- allow users to more easily add NetBox plugins (#530)
- run netbox-initializers plugin on startup even if we're doing a netbox database backup preload (#531)
- during auth_setup "all" operation, do required operations without prompting if the files don't already exist (#536)
- some containers need resource request specified for Kubernetes (#539)
- add "public" pseudo-segments for public IP addresses (#542)
- reworked Windows Event dashboard
- some documentation updates
- added
netbox
tag to any logs that are passed into thenetbox_enrich.rb
script in the Logstash enrichment pipeline
- Component version updates
- elasticsearch and elasticsearch-dsl Python libraries to v8.15.0
- Arkime to v5.4.0
- Beats to v8.15.0
- capa to v7.2.0
- evtx to v0.8.3
- Fluent Bit to v3.1.6
- fluent-bit-setup.ps1 helper script needs updated URLs (#541)
- Logstash to v8.15.0
- NetBox to v4.0.9
- OpenSearch and OpenSearch Dashboards to v2.16.0
- yq to v4.44.3
- Zeek to v7.0.0 (#535)
- Bug fixes
- dashboards-helper container's use of curl fails internal container name resolution when host has invalid DNS settings, prevents Malcolm initialization (#499)
- Netbox service templates not populating (#522)
- kubernetes manifest for netbox refers to netbox-netmap-json configmap which no longer exists (#540)
- don't try to expose the OpenSearch port 9200 in
docker-compose.yml
when the database mode is notopensearch-local
- improved the liveness check for the offline Zeek container so that it returns "healthy" if the intel thread feeds are still pulling before the monitoring processes start up
- missing cracklib-runtime package prevents ISO service account password from being updated by non-root user (#548)
Official ISO installer images for Malcolm and Hedgehog Linux can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split into 2GB chunks and can be reassembled with scripts provided for both Bash (release_cleaver.sh
) and PowerShell (release_cleaver.ps1
). See Downloading Malcolm - Installer ISOs for instructions.
Malcolm v24.07.0
Malcolm v24.07.0 contains minor improvements, some component version updates, and a few bug fixes.
- Features and enhancements
- integrated the ICSNPP GE SRTP network analyzer (#516)
- Changed the way
docker compose
does bind mounts of files and directories to avoid creating empty directories when the source is missing, returning an error instead (#473)- This changed necessitated a switch from Python's built-in YAML library to ruamel.yaml
- code to pull from MISP feeds should specify JSON as preferred format in HTTP headers (#520)
- add optional
service
argument torestart
script (#521) - replace API link on landing page with extracted-files (#524)
- exclude private IP space Intel::ADDR items when populating Zeek intel (#528)
- updated some screenshots for the documentation
- Component version updates
- Bug fixes
- tarball-based installation should not depend on UID inside of tarball, prevents installation if UID with which tarball's contents were created don't match installing user's (#519)
- bacnet discovery log not parsed correctly (#523)
- resolved issue with the
build.sh
helper script when building non-AMD64 Docker images
- Configuration changes (in environment variables in
./config/
)- The variable
ZEEK_DISABLE_ICS_GE_SRTP
has been added tozeek.env
andcontrol_vars.conf
to control enabling the network analyzer for the GE SRTP protocol. It's default value istrue
(indicating that the analyzer is disabled) as it is a somewhat uncommon OT protocol that likely won't be needed by most Malcolm users.
- The variable
- Other
- Removed long-deprecated
net-map.json
file support (#517)
- Removed long-deprecated
Official ISO installer images for Malcolm and Hedgehog Linux can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split into 2GB chunks and can be reassembled with scripts provided for both Bash (release_cleaver.sh
) and PowerShell (release_cleaver.ps1
). See Downloading Malcolm - Installer ISOs for instructions.
Malcolm v24.06.0
Malcolm v24.06.0 contains new features, improvements, component version updates, and a few bug fixes.
NetBox: backwards compatibility-breaking change: This release of Malcolm updates NetBox from v3.6.7 to v4.0.6, for bug fixes, security updates, and requirements for Malcolm to support enrichment with multiple NetBox sites. However, NetBox's built-in migrations do not appear to work handle going from v3.6.7 to v4.0.6. It is likely that if you are using NetBox that you will encounter errors upon updating to this release of Malcolm. Prior to upgrading it is recommended that you navigate to Sites, IPAM > Prefixes, DCIM > Devices, and anywhere else you've populated NetBox data and click Export > All Data (CSV) and save those in case you need to recreate your NetBox inventory after upgrading. Malcolm's NetBox backup and restore will not work in this case. If you find NetBox has data errors after upgrading Malcolm, stop Malcolm and clear the NetBox inventory from your Malcolm installation directory (e.g., rm -rf ./netbox/postgres/* ./netbox/redis/*
), then start Malcolm and recreate your NetBox inventory.
- Features and enhancements
- Support for multiple NetBox sites (#449)
- Malcolm now supports enrichment from a NetBox inventory for asset interaction analysis across multiple sites. The NetBox site can be specified for uploaded PCAP, for a Hedgehog Linux sensor, and for Malcolm live capture.
- JA4+ replaces the JA3 TLS fingerprinting standard from 2017 (see also this blog post) (#419)
- Support uploading Windows Event Log evtx files (#465) and update associated dashboard
- Document using GitHub runners to build Malcolm images (for contributors' guide, #491)
- Generate new forwarder SSL keys on-the-fly when transferring between Malcolm and Hedgehog Linux (#492)
- Incorporate ATT&CK-based Control-system Indicator Detection for Zeek (ACID) (#489), a collection of Operational Techonology (OT) protocol indicators developed to alert on specific ATT&CK for ICS behaviors
- Add platform architecture and machine boot time to Malcolm version API
- Add links to the navigation pane of most dashboards to "other" dashboards for non-network log data (e.g., resource monitoring, Windows Event logs, etc.)
- Support for multiple NetBox sites (#449)
- Component version updates
- Alpine to v3.20 for some of the Docker images
- Beats to v8.14.1
- capa to v7.1.0
- elasticsearch-dsl to v8.13.1
- elasticsearch-py to v8.14.0
- Fluent Bit to v3.0.7
- Logstash to v8.14.1
- NetBox to v4.0.6 (from v3.6.7, #385)
- OpenSearch and OpenSearch Dashboards to v2.15.0
- opensearch-py to v2.6.0
- psutil to v5.9.8
- Supercronic to v0.2.30
- urllib3 to v1.26.19
- yq to v4.44.2
- Bug fixes
- Arkime viewer not rolling PCAPs (#484)
- Free up space in GitHub runner environment building ISO images to avoid errors due to exhausted disk space
- Configuration changes in environment variables
- There are no significant changes or additions to the
./config/*.env
environment variable files in Malcolm v24.06.0
- There are no significant changes or additions to the
Official ISO installer images for Malcolm and Hedgehog Linux can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split into 2GB chunks and can be reassembled with scripts provided for both Bash (release_cleaver.sh
) and PowerShell (release_cleaver.ps1
). See Downloading Malcolm - Installer ISOs for instructions.
Malcolm v24.05.0
Malcolm v24.05.0 contains new features, improvements, bug fixes and component version updates.
- Features and enhancements
- Added ARM64/AArch64 support. Malcolm can now run natively on ARM64 hardware. The
./scripts/configure
script should detect the architecture and automatically adjust theimage:
names in thedocker-compose.yml
files in Docker deployments, or this can be changed manually by appending-arm64
to the tag for Malcolm's Docker images, e.g.,ghcr.io/idaholab/malcolm/zeek:24.05.0-arm64
. (#369) - Support for new environment variables added to Hedgehog Linux's
control_vars.conf
for tuning capture-related settings for Arkime on the sensor, bringing them into parity with those that were available in thearkime-live
container in Malcolm. (#476) - Tweaked some of the default resource-related live capture settings for Suricata and Arkime.
- Reworked the environment variables used for tuning Zeek live capture resource and performance on both Malcolm and Hedgehog Linux. An in-depth discussion of these tuning parameters can be found in the documentation. (#475)
- Allow setting the spiDataMaxIndexes variable for Arkime's
config.ini
file via theARKIME_SPI_DATA_MAX_INDICES
environment variable. (#471) - Allow custom tags to be specified at the point of log file ingestion (i.e., FileBeat) on Malcolm and Hedgehog Linux. This makes it easier to specify custom tags used to group network traffic by sensor. (#463)
- Handle invalid URLs made to the Malcolm web-based UIs better (with a custom 404/502 page). (#461)
- Switched to official .deb packages for Arkime rather than building from source, reducing build times significantly. (Thanks @awick.)
- Added ARM64/AArch64 support. Malcolm can now run natively on ARM64 hardware. The
- Component version updates
- Suricata to v7.0.5
- Also, going forward Malcolm will track the latest Suricata release (from the Debian Stable Backports APT repository) rather than what's in the Debian Stable APT repository. (#462)
- Arkime to v5.2.0
- OpenSearch and OpenSearch Dashboards to v2.14.0
- YARA to v4.5.1
- Beats to v8.13.4
- Logstash to v8.13.4
- YQ to v4.44.1
- Zeek to v6.2.1
- Fluent Bit to v3.0.6
requests
Python library to v2.32.0 for CVE-2024-35195flask-cors
Python library on Hedgehog Linux to v4.0.1 for CVE-2024-1681Jinja
Python library on Hedgehog Linux to v3.1.4 for CVE-2024-34064Werkzeug
Python library on Hedgehog linux to v3.0.3 for CVE-2024-34069
- Suricata to v7.0.5
- Bug fixes
- The code that cleans up already-processed Zeek and Suricata logs after a defined period of time was out of date for the current FileBeat registry behavior and would potentially leave log files around longer than they needed to be. This has been remedied. (#479)
- Fixed issue where the BPF capture filter was not passed to Zeek correctly. (#474)
- The process which queries threat intelligence feeds and generates the corresponding Zeek intel files will no longer relpace existing intel definitions unless it succeeds in pulling definitions from at least one of the specified feeds. (#472)
- Fixed calculation of memory and CPU resources used in
./scripts/status
for Kubernetes deployment. (#467)
- Configuration changes (in environment variables in
./config/
) for Malcolm and incontrol_vars.conf
for Hedgehog Linux- Malcolm
- Added
ARKIME_SPI_DATA_MAX_INDICES
toarkime.env
with a default value of7
, which manifests asspiDataMaxIndexes
in Arkime's config.ini. If you are changing the Arkime index period from daily to weekly, hourly, etc., you may wish to adjust this value. (#471) - Added
EXTRA_TAGS
toupload-common.env
for specifying custom tags to be associated with logs forwarded to Logstash by FileBeat. (#463) - A number of new and modified environment variables are available and can be added to
zeek-live.env
for tuning the resource utilization and performance of Zeek's live capture, see the documentation for details. (#475)
- Added
- Hedgehog Linux
- A number of new and modified environment variables are available for
control_vars.conf
for tuning the resource utilization and performance of Zeek's live capture, see the documentation for details. (#475) - Added support for new environment variables added to Hedgehog Linux's
control_vars.conf
for tuning capture-related settings for Arkime on the sensor, bringing them into parity with those that were available in thearkime-live
container in Malcolm. (#476)
- A number of new and modified environment variables are available for
- Malcolm
Official ISO installer images for Malcolm and Hedgehog Linux can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split into 2GB chunks and can be reassembled with scripts provided for both Bash (release_cleaver.sh
) and PowerShell (release_cleaver.ps1
). See Downloading Malcolm - Installer ISOs for instructions.
Malcolm v24.04.0
Malcolm v24.04.0 contains new features, improvements, bug fixes and component version updates.
Because some of the environment variables used for configuring Malcolm have been reorganized in the .env
files found in the ./config
directory, it is recommended you re-run ./scripts/configure
for this release.
- Features and enhancements
- Zeek-extracted files scanned and preserved on a Hedgehog Linux sensor can now be accessed via the extracted files download user interface (#331).
- Improvements to creation of index templates, dashboards, and other saved objects on startup (#208) to ensure that saved objects get created correctly upon upgrade (see this comment for more details on this feature).
- Populating the NetBox inventory via passively-gathered network traffic metadata now uses network traffic logs for DNS, NTLM, and DHCP to identify assets' host names when possible for use when populating device and VM names (#415). Autopopulated devices now have their status field set to
Active
rather thanStage
, and uses tags instead to indicated that they were created through autopopulation. - Users can now specify pruning thresholds for carved files so that old files are deleted in order to avoid filling available storage (#453). See a new section of documentation on Managing disk usage for more information about this and similar settings.
- Users can now specify a prefix that will be prepended to dashboards as they are imported into OpenSearch Dashboards or Kibana, allowing users who have dashboards from other sources to differentiate between those and Malcolm's (#455).
- The default anomaly detectors created for the OpenSearch Anomaly Detection plugin are now created with category fields for high cardinality to allow for better breakdown of contributing values to anomalies discovered (#464).
- Include JA4+ plugin in Arkime. See #419 for status on upcoming full JA4+ support in Malcolm.
- Hedgehog Linux sensors can now periodically refresh their Zeek inteligence files.
- NOTE: Due to an oversight, a value is missing from the default Hedgehog Linux configuration in this release, preventing the intel refresh cron job from executing. As a workaround, appending the line
export INTEL_DIR=/opt/sensor/sensor_ctl/zeek/intel
to/opt/sensor/sensor_ctl/control_vars.conf
and restarting the sensor services will remedy the situation. This will be corrected in the next Malcolm release.
- NOTE: Due to an oversight, a value is missing from the default Hedgehog Linux configuration in this release, preventing the intel refresh cron job from executing. As a workaround, appending the line
- Assorted documentation improvements.
- Component version updates
- Arkime to v5.1.2
- OpenSearch and OpenSearch Dashboards to v2.13.0
- Beats to v8.13.2
- Logstash to v8.13.2
- gunicorn to v22.0.0 to address CVE-2024-1135.
- elasticsearch-dsl to v8.13.0
- elasticsearch-py to v8.13.0
- idna to v3.7 to address CVE-2024-3651
- Fluent Bit to v3.0.3
- Bug fixes
- The documentation for Windows host system configuration was out of date and has been updated for the latest version of Microsoft Windows Subsystem for Linux (#421).
- An issue was fixed in which Malcolm's list of users and their password hashes could become corrupted if the file did not initially end with a newline character (#426).
- The manner in which Zeek intel files are generated has been changed to avoid problems found in Kubernetes deployments when scaling out the number of
zeek-live
containers (#456). See this comment for more details. - Removed the version top-level element from
docker-compose.yml
files as it is now obsolete and caused a warning message that sometimes was not handled correctly. - Fix Malcolm ISO not correctly detecting if it's in a live boot ISO environment or installed mode.
- Restart live Zeek instances with
zeekctl deploy
instead ofzeekctl restart
.
- Configuration changes (in environment variables in
./config/
)ARKIME_QUERY_ALL_INDICES
inarkime.env
can be set to control thequeryAllIndices
setting in Arkime'sconfig.ini
.DASHBOARDS_PREFIX
indashboards-helper.env
has been added for #455 (see above in Features and Enhancements).LOGSTASH_NETBOX_ENRICHMENT_DATASETS
inlogstash.env
has been changed to includezeek.dhcp
,zeek.dns
, andzeek.ntlm
to support #415 (see above in Features and Enhancements).LOGSTASH_ZEEK_IGNORED_LOGS
inlogstash.env
has been changed to removecapture_loss
andstats
so that those diagnostic Zeek logs can be parsed without the user having to manually change this variable.ZEEK_CRON
has been removed fromzeek-live.env
andZEEK_INTEL_REFRESH_CRON_EXPRESSION
was removed fromzeek.env
and moved to the "offline" version of the container inzeek-offline.env
for #456.EXTRACTED_FILE_PRUNE_THRESHOLD_MAX_SIZE
,EXTRACTED_FILE_PRUNE_THRESHOLD_TOTAL_DISK_USAGE_PERCENT
, andEXTRACTED_FILE_PRUNE_INTERVAL_SECONDS
were added tozeek.env
for #453. See a new section of documentation on Managing disk usage for more information about these and similar settings.
Official ISO installer images for Malcolm and Hedgehog Linux can now be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split into 2GB chunks and can be reassembled with scripts provided for both Bash (release_cleaver.sh
) and PowerShell (release_cleaver.ps1
). See Downloading Malcolm - Installer ISOs for instructions.