Publishing to Firefly-iii using Pandas

This article is part of the Self-hosted Finances series.

    Previously, in my Self-hosted finances series, I cleaned and identified transfers in my Mint transactions for the purposes of of importing into Firefly-iii. In this post, I’m going to import the transactions into Firefly-iii.

    This part is comparatively easy vs the previous steps, however it’s only a one time import. A continue updating workflow is tricky and I’m working on some logic to do that.

    Authenticating

    First, we need a token that allows us to call the API. To get this, login to Firefly, and open Options, then Profile.

    Then go to OAuth and click Create a new token

    A screenshot of FIrefly showing the options page and the OAuth client page. Firefly will then show you a token. Copy and store this in the script as a variable and also specify the endpoint where Firefly is being served:

    1
    2
    
    fireflyToken = 'eYJ...'
    host = 'https://firefly.example.com'
    

    Constructing the request

    This next method will translate the DataFrame created in the previous post request into the format that the Firefly API expects and creates the transaction.

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    
    import json
    import requests
    
    session = requests.Session()
    
    def send_to_firefly(record):
        item = {
            'type': record['type'],
            'amount': record['amount']
        }
        for key in ['description',  'destination_name', 'source_name', 'category_name', 'notes']:
            if key in record and not pd.isnull(record[key]):
                item[key] = record[key]
        for key in ['source_id', 'destination_id']:
            if key in record and not pd.isnull(record[key]) and record[key] != '':
                item[key] = str(int(record[key]))
        item['foreign_amount'] = '0'
        item['reconciled'] = False
        item['order'] = '0'
        # If you use something other than USD, change this value
        item['currency_id'] = '8'
        item['currency_code'] = 'USD'
        for key in ['date', 'process_date']:
            if key in record and not pd.isnull(record[key]) and record[key] != '':
                item[key] = record[key].isoformat()
        if isinstance(record['tags'], str):
            item['tags'] = [record['tags']]
        else:
            item['tags'] = []
        payload = {
                "transactions": [
                    item
                ],
                "apply_rules": True,
                "fire_webhooks": False,
                "error_if_duplicate_hash": False
            }
        try:
            json.dumps(payload)
        except Exception as e:
            raise Exception(f"Can't serialize '{item}: {e}")
        resp = session.post(f"{host}/api/v1/transactions", headers={"Authorization": f"Bearer {fireflyToken}", "Content-Type": "application/json", 'Accept': 'application/json'}, json=payload, allow_redirects=False)
        if resp.status_code == 200:
            return pd.Series([resp.status_code, resp.json(), resp.json()['data']['id']], index=['status', 'message', 'firefly_id'])
        else:
            return pd.Series([resp.status_code, resp.json()], index=['status', 'message'])
    

    Sending the request

    Next, it’s easy to send the requests to Firefly:

    1
    2
    3
    4
    5
    
    # First find the transfers and translate into common format
    output = transactions.apply(func=process_record, axis=1, result_type='expand')
    
    # Then send it to Firefly
    fireflyResult = output[output['amount'] != 0].apply(func=send_to_firefly, axis=1, result_type='expand')
    

    Reviewing Results

    Firefly can return non-200 status code responses. Running this will identify all the non-successful writes. In theory, this should empty. It’s a good idea to check the error messages and see what’s wrong.

    1
    
    print(fireflyResult[(fireflyResult['status'] != 200)])
    

    Conclusion

    Once we’ve identified the transfers, it’s comparatively easy to write the one-time extract to Firefly. The code can be found in my GitHub repository. I’ve got a few projects in progress right now, including tools to automatically scrape data from accounts and how to merge new transactions into Firefly.

    Copyright - All Rights Reserved

    Comments

    To give feedback, send an email to adam [at] this website url.

    Other Posts in Series

    This post is part of the Self-hosted Finances series. You can check out the other posts for more information:

      Donate

      If you've found these posts helpful and would like to support this work directly, your contribution would be appreciated and enable me to dedicate more time to creating future posts. Thank you for joining me!

      Donate to my blog