Prev vacancy

Data Engineer - SC Cleared

Next vacancy
  • Remote working
  • to £600 per day
  • Contract , 6 months
  • Full time
  • Industry: Government, IT & Telecoms
  • Discipline: Data Analyst, Data Management, Data modelling, Database analysis, Cloud

Our client a central government department are currently looking for a Data Engineer with Active SC Clearance to join their team on a 6 months contract based in London/WFH paying £600pd inside IR35.

Technical Skills required

This appointment is expected to form an initial part of a longer term programme of work, therefore significant progress is expected to made against the following deliverables (on some cases dependent upon prioritisation of development effort):

• Translate business requirements to ensure they are clear and robust, and can be transformed into reusable production ready code and / or effective data models;

• Define version control mechanisms and strategies for the source code of products;

• Support Quality Assurance colleagues and Data Managers responsible for the day to day testing and live running of data products and services;

• Shape solutions in a fit for purpose way; following the agreed principles and contribute to the overall Data Engineer Development lifecycle;

• Perform data profiling and quality measurements;

• Designing and developing reusable metadata libraries;

• Owning the stability of new products designed, including the on-going robustness, resilience and stability of these products;

• Identifying, managing and resolving issues preventing the delivery or continuous development of products

Specific to Big Data Developer role:

• Responsible for delivering data processing pipelines in a cloud or on premise environment using distribute file store technology.

• Responsible for creating and maintaining data pipelines using data streaming or batch transfer technology

• Designing ELT processes, data loading strategy, data archiving, data reconciliation, data manipulation, data integration, error handling and error logging mechanisms.

Knowledge and Experience required:

• Proven experience of big data engineering techniques and concepts using the Hadoop Stack (Cloudera/EMR), including data ingestion, processing and storage using HDFS, Spark, Hive and Impala.

• Extensive, hands-on experience of large complex Data Engineering projects designing and developing ETL pipelines in a cloud or on premise environment

• Experience of design and implementation of data storage, including HDFS, S3, relational and NoSQL

• Experience of developing/utilising programming and query languages e.g. SQL, Java, Scala

• Monitoring performance and advising on any required infrastructure or changes.

• A good understanding of data management, governance and quality frameworks, and how these integrate with big data solutions

CV 24th November at 12pm

  • Quote ref: 990035/21943

Similar jobs

Enterprise Architect
Department for Transport (DFT), Bristol, UK
from £700 to £730 per day, Contract

Enterprise Architect

Our client a public sector department are currently looking for an Enterprise Architect to join their team on a 1 year...
.Net/SQL Developer - Java experience
UK Central Government, Taunton, UK
to £500 per day, Contract

.Net/SQL Developer - Java experience

Our client a central government department are currently looking for an SC Cleared Developer to join their team on a...
This website uses cookies to enhance your user experience. By continuing to use this site, you consent to our use of these cookies. See our Cookie Policy.