Our client a central government department are currently looking for a Data Engineer with Active SC Clearance to join their team on a 6 months contract based in London/WFH paying £600pd inside IR35.
Technical Skills required
This appointment is expected to form an initial part of a longer term programme of work, therefore significant progress is expected to made against the following deliverables (on some cases dependent upon prioritisation of development effort):
• Translate business requirements to ensure they are clear and robust, and can be transformed into reusable production ready code and / or effective data models;
• Define version control mechanisms and strategies for the source code of products;
• Support Quality Assurance colleagues and Data Managers responsible for the day to day testing and live running of data products and services;
• Shape solutions in a fit for purpose way; following the agreed principles and contribute to the overall Data Engineer Development lifecycle;
• Perform data profiling and quality measurements;
• Designing and developing reusable metadata libraries;
• Owning the stability of new products designed, including the on-going robustness, resilience and stability of these products;
• Identifying, managing and resolving issues preventing the delivery or continuous development of products
Specific to Big Data Developer role:
• Responsible for delivering data processing pipelines in a cloud or on premise environment using distribute file store technology.
• Responsible for creating and maintaining data pipelines using data streaming or batch transfer technology
• Designing ELT processes, data loading strategy, data archiving, data reconciliation, data manipulation, data integration, error handling and error logging mechanisms.
Knowledge and Experience required:
• Proven experience of big data engineering techniques and concepts using the Hadoop Stack (Cloudera/EMR), including data ingestion, processing and storage using HDFS, Spark, Hive and Impala.
• Extensive, hands-on experience of large complex Data Engineering projects designing and developing ETL pipelines in a cloud or on premise environment
• Experience of design and implementation of data storage, including HDFS, S3, relational and NoSQL
• Experience of developing/utilising programming and query languages e.g. SQL, Java, Scala
• Monitoring performance and advising on any required infrastructure or changes.
• A good understanding of data management, governance and quality frameworks, and how these integrate with big data solutions
CV 24th November at 12pm