share a transform by multiple entities - multiple input_type

[cvandeplas]

Following the documentation I understand we can use inheritance to re-use code when different entities use the same logic. The input_type field defines which entity is allowed to use this code.
This works indeed quite well.

class HashToEvent(AttributeToEvent):
    input_type = Hash

class DomainToEvent(AttributeToEvent):
    input_type = Domain

class IPv4AddressToEvent(AttributeToEvent):
    display_name = 'IPv4AddressToEvent'
    input_type = IPv4Address

However the downside is that this limits running a transform when the user selects multiple entities and wants to run the same transform in one click.

I have tried to set the display_name to the same thing, however this doesn't really result in the hoped effect:

I have also tried to do put a list in the input_type = [Domain, IPv4Address] but that does not work.

Is there a way to assign multiple entities to one single transform? Or ensure one transform is created in the Transforms pane?
Thanks

[cvandeplas]

While there is no(t yet a) way to allow a transform to be only used by a few entities, it is possible to have it available for all the entities.

The documentation states:

At the top of the tree is ‘maltego.Unknown’. This means that if you configure a transform to run on this base entity type – it will be available when you right click … on any entity.

Unfortunately setting the input_type to Unknown does not work, as canari creates the entity as canari.Unknown when imported in Maltego.

However a thing that works is to create your own custom entity:

class Unknown(Entity):
    _category_ = 'Unknown'
    _namespace_ = 'maltego'

Then import your Unknown and create your transform:

from <your_project>.transforms.common.entities import Unknown
class MyTransform(Transform):
    input_type = Unknown

    def do_transform(self, request, response, config):
        pass  # do magic here

[maxlandon]

It would indeed be fantastic if there was support for multiple entities as input for transforms. I also really need it, and it seems many will need it if they get to build complex entity sets along with their transforms.

Some have tried an even weirder approach, which is to re-implement (yes) a Maltego transform on top of the Canari framework. At least I guess they did it in that goal (Msploitego project).

Thanks for sharing your hack on this, however I can foresee the coming mess when you have 200 different entities on your graph, and 10 transforms for each....

Waiting for news from the Canari maintainer, he might have an idea !

[allfro]

You can subclass the transform and set the entity input type without having to rewrite your do_transform method

[maxlandon]

I've actually tried to subclass the same transform on 3 different entities (therefore ending up with 4 different classes). I doesn't work correctly for me. Let me try again and I'lll come back to you.

[maxlandon]

Okay I had done something wrong, sorry for the useless complain...

[maxlandon]

Anyway thanks a zillion times Allfro, I'm having so much fun with your framework. Lots of things to be done with it !

[allfro]

Glad to hear 😊

[allfro]

@cvandeplas:

Regarding multiple entity type selection: I see what you mean. I think this is a problem with the Maltego client more than the framework itself. We can try to add workarounds in the framework but that may have adverse affects to the client-side in future releases. I have seen instances where this was not an issue in older versions of the client. However, this newer version is not as stable. I would open a ticket with Paterva for this.

[maxlandon]

@cvandeplas, if you are kind enough to open a ticket with Paterva concerning this issue, could you keep us informed on this thread please ?
If this is solved, everything in this world would be perfect !
Looking forward to ear from you !

[maxlandon]

Hello @cvandeplas @allfro, I again took a look at this issue of having multiple transforms with the same name when selecting different entities.
I found the solution in the Paterva's documentation:

Entity Inheritance - It is strongly recommended that if you do have entities that could be related to entities pre-existing in the tool that you use entity inheritance to avoid having to duplicate transforms. An example of this is the website, NS record and MX record that all inherit from the DNS entity. This means that instead of having to recreate the 'to IP Address' transform for each type we create it only once for the DNS record entity and the transform will be available to all of the different types.

So, if you correctly use the fully qualified type of your Entities (so that you make a consistent Entity tree), you.... DON'T HAVE TO SUBCLASS THE TRANSFORMS IN YOUR PYTHON CODE !

(All of this sounds like you need to go back and forth between the Maltego client and the Canari Framework, understand all the implications of the namespaces and types, but at the end, everything runs fine and is easy to maintain !)
Take a look at the 3 issues (tagged questions, now closed) where @allfro details the implications of using namespace and type attributes, you will find this very helpful !

[cvandeplas]

@maxlandon thanks for the feedback. To my understanding this does not resolve the issue when using the default maltego.* entities. The inheritance is described in the entity definition. So this could work for custom entities, but not for default ones.

Also, an inheritance model is great to share transforms, but remains limiting on the level of flexibility.

To me, the ideal way would be to allow multiple entities in the InputConstraints.
Actually, if you open the XML of your foo.transform you will notice:

	<InputConstraints>
		<Entity max="1" min="1" type="maltego.Unknown"/>
	</InputConstraints>

You can edit this and add multiple entries:

	<InputConstraints>
		<Entity max="1" min="1" type="maltego.Domain"/>
		<Entity max="1" min="1" type="maltego.IPv4Address"/>
	</InputConstraints>

If you re-import this transform it will be accepted.
The only downside is that the transform only activates if Entities of all the InputConstraints. So in this case you will not see your transform if you only select an Entity of Domain. But if you select two or more entities, and your selection contains both types, then your transform will be shown in the menu.

[maxlandon]

@cvandeplas thanks for the feedback. I'll update this thread if there are "news from the frontline" concerning this !

[allfro]

I have several comments to add to this thread: 1. Entity inheritance doesn’t always make sense. For example, if you have a transform that sends different kinds of entities to an API (say Person and hash value) then subclassing person or hash doesn’t make sense. But inheritance for something like person and owner does make sense. So using entity inheritance isn’t the silver bullet. You have to use both transform and entity inheritance. 2. I spent a lot of time playing with Maltego and fuzzing the profile when building canari to see what it’s capabilities are and believe me when I say that a lot of the things you are trying have been done before and with mixed results. Mixed because each version of Maltego has interpreted the profile differently. In some cases it completely corrupts Maltego’s base profile and requires a delete of the config folder. So I would completely avoid doing any direct editing if you want to things to work across versions. 3. Subclassing a transform and providing it with the same display name is as easy as setting the `display_name` class property on the subclassed transform to match the original. ```python class ParentTransform(Transform): input_type = Person class ChildTransform(ParentTransform) display_name = “Parent Transform” input_type = Email ``` The reason there is a one to one relationship between the transform input type and transform is because local transforms don’t communicate the input entity type to transforms. So you would have had to work with dictionaries instead of Entity objects. Yuck! I wanted to abstract the entire underbelly of Maltego and it’s api so that developers can focus on writing code and not config files and XML. So this is why multiple entity types are not allowed for one transform.

[maxlandon]

@allfro thanks a lot for these explanations.
I am actually working on this very same use case: making transforms that get/post objects from the Metasploit DB API. The purpose was to have the full versatility of the Msf DB in Maltego, but with the Input Entities acting as a workflow filter.

So:
Because the API query model allows much more granularity than SQL queries, I had to devise how to categorize transforms and their inputs. So I had to say "A host has a transform category that enumerates things lower in its Metasploit hierarchy, another category for push/pull its own properties, etc..."

I ended up using the Transform inheritance the same way you put it in your message.
If your code base is well structured/defined:
1) The categorization of transforms in Maltego is perfectly fine.
2) The Transform inheritance is also perfect, as long as the Entity inheritance is well devised.

I'll add a very last thing:
It is indeed important to consider Maltego Transform inheritance, Maltego Entity inheritance and Canari (Python) code inheritance as 3 distinct things. Why ?
1) Because they can cross each other if circumstances ask for it.
2) Because it offers an amazing level of granularity to our Transform projects, which is indeed wonderful when making transforms interacting with APIs.

The illustration of this message (and of the ones above as well) can be seen by looking/using my project at https://github.com/maxlandon/EffectiveCouscous/tree/v0.2.
The project's documentation also crosses all of what has been said here...

I hope this will be as useful to others as it has been to me !

[maxlandon]

Hello @allfro , two new findings (of which one enormous error of mine) I made today, maybe you can relate to them:

1) Confusing Python code Entity inheritance and Maltego code inheritance.
Python Entity definition class: Do not make an Entity class inherit from the class of an Entity higher hierarchically in Maltego, for the sake of saving a few lines of properties' code...

class MetasploitHost(Entity):
      _namespace = "foo.host"

      prop_one = StringEntityField("")
      prop_two = StringEntityField("")

class LinuxHost(MetasploitHost):
    _namespace_ = "foo.host.MetasploitHost"

""" Using inheritance to get/set props on this Entity class """

...And then accessing the values of LinuxHost() in transforms....

host = LinuxHost()
host.name = "Foo"

... because:
The way Canari looks up class properties, in this case, is ambiguous to the Maltego client (it considers the properties to be MetasploitHost's, not LinuxHost's ones) so it will mess everything up WITH HUNDREDS OF EXCEPTIONS AND A TOTALLY UNUSABLE MALTEGO CLIENT.
It did it, I had to reinstall everything, a huge mess.

Solution: There are 2 things to do (until proven wrong, both of them need to be done):

• In Python, do a copy-paste of the properties, it costs only ten lines, and zero headakes.
• In Maltego Client, go to Entities Manager, and set defaults (-, for instance), that will make these properties appear at all times.

2) Random Maltego Client bug
A few moments later the 1st point, maybe because of an inconsistent Profile or Entity file being imported, Maltego Client did not infer the right hierarchy of Entities (despite the code was right everywhere), so that transforms are not available to all expected entities (between other problems).
I just had to reimport a newly generated Canari profile and the issue disappeared.

Finally
I have been throwing at you at least a dozen of remarks and questions that are not properly ordered, and I may doubt that someone exterior to the conversation will find these useful. I'll grab my courage with both hands soon and try to write a more structured text compiling all the findings of this thread that are undocumented in Canari now.