Mist Catalog

S3Store command

Description

This command stores the data received as lines in the given S3 URI. This command is intended to be used with a stream.

Input parameters

• text: Source stream to read text lines from
• remoteUri: S3Uri of the remote object to create. Must contain bucket, prefix and object name

Output parameters

None.

Dependencies

This command requires the AWS CLI and the credentials configuration in place.

Examples

Store the data received from the stream :source as s3://myBucket/facts/list.txt.

S3Store(:source, "s3://myBucket/facts/list.txt")

Auxiliar functions

S3Writer

This function copies a local file in the given S3Uri.

Input parameters

• localPath: Path of the local file to copy
• remoteUri: S3Uri of the remote object to create

festin command

Description

This command finds open S3 buckets from an origin domain list.

Concurrency Type

Sync or Async

Input parameters

• originDomain: Origin domain to search S3 buckets.
• dns: Dns server to use during execution.
• tor: “True” if you want to use Tor network, “False” otherwise.

Output parameters

• result: Boolean, possible values are: True, if the festin command has been executed without errors, or False otherwise.
• resultCode: Integer. Exit code from festin command.
• consoleOutput: Raw text with console output from festin command.
• consoleError: Raw text with console error from festin command.
• buckets: A list of domains, S3 bucket names and objects found.

NOTE: when used as a producer for a queue (Async) every domain found will be sent as soon as found it.

Tools and services

The following commands need to be available in your command path:

• festin

Example

Find S3 buckets from “wordpress.com” using Tor network and 212.166.64.1 as dns server.

include "festin"
r = festin("wordpress.com", "212.166.64.1", True)
print(r["buckets"])

filterRepeated command

Description

To use with queues. This command filter repeated values in a given list. And send non repeated values.

Input parameters

• v: Any. Input value to check
• values: List where all values will be saved. (With no repetitions)

Output parameters

Non repeated items will be sent.

Examples

Filer repeated domains at input queue “domains” and output non repeated to queu “nonRepeatedDomains”.

domainProcessed = []
filterRepeated(:domains, domainProcessed) => nonRepeatedDomains

findOpenPorts command

Description

This command performs a port scan to a target host.

Concurrency Type

Sync or Async

Input parameters

• ip: String. Target IP or dns name.
• ports: String. Ports to scan, one, a range or a list of ports giving, optionally, the protocol (U=UDP, T=TCP, S=SYN scan). Ex: “22”, “1-65535”, “U:53,111,137,T:21-25,80,139,8080,S:9”.

Output parameters

• result: Boolean, possible values are: True, if the nmap command has been executed without errors, or False otherwise.
• resultCode: Integer. Exit code from nmap command.
• consoleOutput: Raw text with console output from nmap command.
• consoleError: Raw text with console error from nmap command.
• openPorts: A list of open ports found.

NOTE: when used with a queue the list of openPorts will be send with the format: {“ip”: “1.2.3.4”, “port”: 443, “protocol”: “tcp”}

Tools and services

The following commands need to be available in your command path:

• nmap

Examples

Find open ports among a range (0-9000) at localhost and print them out on console.

include "findOpenPorts"
r = findOpenPorts("127.0.0.1", "0-9000")
print(r)

Expected output:

{'result': True, 'resultCode': 0, 'consoleOutput': 'Starting Nmap 7.80 ( https://nmap.org ) at 2021-02-17 11:48 CET\nNmap scan report for localhost (127.0.0.1)\nHost is up (0.00017s latency).\n\nPORT     STATE SERVICE\n631/tcp  open  ipp\n8021/tcp open  ftp-proxy\n\nNmap done: 1 IP address (1 host up) scanned in 0.03 seconds', 'consoleError': '', 'openPorts': [{'port': '631', 'protocol': 'tcp'}, {'port': '8021', 'protocol': 'tcp'}]}

gitLeaksFinder command

Description

Finds sensitive information in Git repositories.

Concurrency Type

Sync

Input parameters

• gitPath: Path of Git repositories.

Output parameters

• result: Boolean, possible values are: True, if the gitleaks command has been executed without errors and not issues detected, or False otherwise.
• resultCode: Integer. Exit code from gitleaks command.
• consoleOutput: Raw text with console output from gitleaks command.
• consoleError: Raw text with console error from gitleaks command.
• issues: List of issues found by analyzer.

Issues contains an array of objects with the following information:

  [
    {
      "commit": {COMMIT-ID},
      "repo": {REPO NAME},
      "rule": {RULE NAME},
      "line": {LINE NUMBER}
    }
  ]

Tools and services

The following commands need to be available in your command path:

• gitleaks

Examples

Find sensitive information in git repository at current directory

include "gitLeaksFinder"
r = gitLeaksFinder("./")
print(r)

Example output:

{'result': False, 'resultCode': 1, 'consoleOutput': '\x1b[36mINFO\x1b[0m[2021-02-17T16:46:02+01:00] report written to /var/folders/vt/0gyg50fx3_q3nn0dwhmjgynh0000gp/T/tmpuwfs8mvr\n\x1b[33mWARN\x1b[0m[2021-02-17T16:46:02+01:00] 1 leaks detected. 9 commits scanned in 1 second 738 milliseconds 202 microseconds', 'consoleError': '', 'issues': [{'line': '          "current_key": "XXXXXXXXXXXXXXXXXXX"', 'lineNumber': 24, 'offender': 'XXXXXXXXXXXXXXXXXXX', 'commit': 'XXXXXXXXXXXXXXXXXXX', 'repo': 'XXXXXXXXXXXXXXXXXXX', 'rule': 'Google API key', 'commitMessage': 'Version 1.0.2\n', 'author': 'XXXXXXXXXXXXXXXXXXX', 'email': 'XXXXXXXXXXXXXXXXXXX', 'file': 'google-services.json', 'date': '2019-08-25T14:14:34+02:00', 'tags': 'key, Google', 'operation': 'addition'}]}

kafkaConsumer command

Description

Read from a Kafka topic and send values to a queue.

Concurrency Type

Async. This function have to be alwais used with a target queue.

Input parameters

• servers: String. Server list for bootstrap-server options. Example: “127.0.0.1:9092”
• topic: String. Kafka topic to listen.
• endMessage: String. When this message arrives the consumer will terminate.
• fromBeginning: Boolean. True for read the topic from the beginning.

Output parameters

Every message received in the topic will be send to the queue.

Tools and services

The following commands need to be available in your command path:

• kafka-console-consumer

Examples

Connect to Kafka at 127.0.0.1:9092, listen the topic “myTopic” for new messages and send then to “myQueue”. When the message “END” arrives, stop listening and finish the process.

include "kafkaConsumer"

function echo(s) {
    print(s)
}

kafkaConsumer("127.0.0.1:9092", "prueba", "END", True) => myQueue
echo(:myQueue)

kafkaProducer command

Description

Write messages to a Kafka topic

Concurrency Type

Sync(for one message) or Async(for muliple messages received in a queue)

Input parameters

• message: String. The message to send. You may also want to use a source queue here.
• servers: String. Server list for bootstrap-server options. Example: “127.0.0.1:9092”
• topic: String. Kafka topic to send messages.

Output parameters

Every “message” received will be send to kafka topic.

Tools and services

The following commands need to be available in your command path:

• kafka-console-producer

Examples

Async. Connect to Kafka at 127.0.0.1:9092, listen the topic “q” for new messages and send then to Kafla topic “prueba”.

include "kafkaProducer"

kafkaProducer(:q, "127.0.0.1:9092", "prueba")

send("msg1","q")
send("msg2","q")

Sync. Connect to Kafka at 127.0.0.1:9092 and send the message “hello” to Kafla topic “prueba”.

include "kafkaProducer"

kafkaProducer("hello", "127.0.0.1:9092", "prueba")

mongoWatch command

Description

Subscribe to a MongoDB collection for new inserts. Every time that a document is inserted in the collection it will be send for the output stream This command run for ever. If you want to stop the program you will need to do Control-C or programatically call to abort function

NOTE: Due to MongoDB limitations the database must be a replicaset. An stand alone database will not work.

Concurrency Type

Async

Input parameters

• uri: MongoDB URI including database. i.e.: mongodb+srv://cluster0.ou8h3.mongodb.net/myDatabase
• user: Username for MondoDB connection
• password: Password for MondoDB connection
• collection: Collection to watch for new inserts.

Output parameters

The document inserted. Output stream.

Tools and services

The following commands need to be available in your command path:

• mongo

Example

Use $MONGO… enrivonment variables to connect to MongoDB.
Watch “customers” collection.
Print every new document inserted.

include "mongoWatch"

mongoWatch($MONGO_URI, $MONGO_USER, $MONGO_PASSWORD, "customers") => doc

doc => print()

PythonCodeAnalysis command

Description

Performs security code analysis on Python code.

Concurrency Type

Sync

Input parameters

• sources: Path of Python source code.

Output parameters

• result: Boolean, possible values are: True, if the gitleaks command has been executed without errors and not issues detected, or False otherwise.
• resultCode: Integer. Exit code from gitleaks command.
• consoleOutput: Raw text with console output from gitleaks command.
• consoleError: Raw text with console error from gitleaks command.
• issues: List of issues found by the analyzer.

Tools and services

The following commands need to be available in your command path:

• bandit

Examples

Basic: Find vulnerabilities in python code ay current directory

include "gitLeaksFinder"
r = gitLeaksFinder("./")
print(r)

searchDomains command

Description

This command performs a certificate search at https://crt.sh/ for an origin domain in order to find other related domains.

Concurrency Type

Sync or Async

Input parameters

• originDomain: Origin domain to search for at https://crt.sh/.

Output parameters

• result: Boolean, possible values are: True, if the dnsrecon command has been executed without errors, or False otherwise.
• resultCode: Integer. Exit code from dnsrecon command.
• domains: A list of domains found. This output will also be send to a queue if requiered.
• consoleOutput: Raw text with console output from dnsrecon command.
• consoleError: Raw text with console error from dnsrecon command.

NOTE: when used as a producer for a queue (Async) every domain found will be sent as soon as found it.

Tools and services

The following commands need to be available in your command path:

• dnsrecon

Example

Find related domains for “bbva.com” and print result

include "searchDomains"
r = searchDomains("bbva.com")
print(r)

tail command

Description

This command is a wrapper for tail command. It will read a file line by line and send each line to a queue. The command will terminate when read an specific EOF token.

Concurrency Type

Async

Input parameters

• file: String. File to read.
• endline: String. EOF line token.

Output parameters

Each line will be send to the target queue

Tools and services

The following commands need to be available in your command path:

• tail

Example

Read file domains.txt, send every line to queue “l”. Stop reading when “END” is read.

include "tail"
tail("domains.txt","*END*") => l
print(: l)

trello library

Description

This command allow to add a new card to a trello list easily. It is needed to set the following environment variables:

• TRELLO_API_KEY
• TRELLO_TOKEN

Please, see Trello API documentation to get those values (https://trello.com/app-key)

Concurrency Type

Sync

Input parameters

• idList: String. List id for add the new card
• name: String. Card title.
• desc: String. Card description.

Output parameters

curl execution result

Tools and services

The following commands need to be available in your command path:

• curl

Example

Create a new card at list with id 609cf4drtd25e143c94afd42. Please, see Trello API documentation to get this value (https://developer.atlassian.com/cloud/trello/rest/api-group-actions/)

include "trelloNewCard"
trelloNewCard("609cf4drtd25e143c94afd42", "Test Card", "Mi test card description")