Google Cloud Certified Professional Data Engineer

Designing knowledge processing techniques

Deciding on the suitable storage applied sciences. Issues embrace:

●  Mapping storage techniques to enterprise necessities

●  Information modeling

●  Commerce-offs involving latency, throughput, transactions

●  Distributed techniques

●  Schema design

Designing knowledge pipelines. Issues embrace:

●  Information publishing and visualization (e.g., BigQuery)

●  Batch and streaming knowledge (e.g., Dataflow, Dataproc, Apache Beam, Apache Spark and Hadoop ecosystem, Pub/Sub, Apache Kafka)

●  On-line (interactive) vs. batch predictions

●  Job automation and orchestration (e.g., Cloud Composer)

Designing an information processing resolution. Issues embrace:

●  Selection of infrastructure

●  System availability and fault tolerance

●  Use of distributed techniques

●  Capability planning

●  Hybrid cloud and edge computing

●  Structure choices (e.g., message brokers, message queues, middleware, service-oriented structure, serverless features)

●  At the very least as soon as, in-order, and precisely as soon as, and so forth., occasion processing

Migrating knowledge warehousing and knowledge processing. Issues embrace:

●  Consciousness of present state and how you can migrate a design to a future state

●  Migrating from on-premises to cloud (Information Switch Service, Switch Equipment, Cloud Networking)

●  Validating a migration

Constructing and operationalizing knowledge processing techniques

Constructing and operationalizing storage techniques. Issues embrace:

●  Efficient use of managed companies (Cloud Bigtable, Cloud Spanner, Cloud SQL, BigQuery, Cloud Storage, Datastore, Memorystore)

●  Storage prices and efficiency

●  Life cycle administration of knowledge

Constructing and operationalizing pipelines. Issues embrace:

●  Information cleaning

●  Batch and streaming

●  Transformation

●  Information acquisition and import

●  Integrating with new knowledge sources

Constructing and operationalizing processing infrastructure. Issues embrace:

●  Provisioning assets

●  Monitoring pipelines

●  Adjusting pipelines

●  Testing and high quality management

Operationalizing machine studying fashions

Leveraging pre-built ML fashions as a service. Issues embrace:

●  ML APIs (e.g., Imaginative and prescient API, Speech API)

●  Customizing ML APIs (e.g., AutoML Imaginative and prescient, Auto ML textual content)

●  Conversational experiences (e.g., Dialogflow)

Deploying an ML pipeline. Issues embrace:

●  Ingesting applicable knowledge

●  Retraining of machine studying fashions (AI Platform Prediction and Coaching, BigQuery ML, Kubeflow, Spark ML)

●  Steady analysis

Selecting the suitable coaching and serving infrastructure. Issues embrace:

●  Distributed vs. single machine

●  Use of edge compute

●  {Hardware} accelerators (e.g., GPU, TPU)

Measuring, monitoring, and troubleshooting machine studying fashions. Issues embrace:

●  Machine studying terminology (e.g., options, labels, fashions, regression, classification, suggestion, supervised and unsupervised studying, analysis metrics)

●  Affect of dependencies of machine studying fashions

●  Widespread sources of error (e.g., assumptions about knowledge)

Guaranteeing resolution high quality

Designing for safety and compliance. Issues embrace:

●  Id and entry administration (e.g., Cloud IAM)

●  Information safety (encryption, key administration)

●  Guaranteeing privateness (e.g., Information Loss Prevention API)

●  Authorized compliance (e.g., Well being Insurance coverage Portability and Accountability Act (HIPAA), Youngsters’s On-line Privateness Safety Act (COPPA), FedRAMP, Normal Information Safety Regulation (GDPR))

Guaranteeing scalability and effectivity. Issues embrace:

●  Constructing and working check suites

●  Pipeline monitoring (e.g., Cloud Monitoring)

●  Assessing, troubleshooting, and enhancing knowledge representations and knowledge processing infrastructure

●  Resizing and autoscaling assets

Guaranteeing reliability and constancy. Issues embrace:

●  Performing knowledge preparation and high quality management (e.g., Dataprep)

●  Verification and monitoring

●  Planning, executing, and stress testing knowledge restoration (fault tolerance, rerunning failed jobs, performing retrospective re-analysis)

●  Selecting between ACID, idempotent, ultimately constant necessities

Guaranteeing flexibility and portability. Issues embrace:

●  Mapping to present and future enterprise necessities

●  Designing for knowledge and utility portability (e.g., multicloud, knowledge residency necessities)

●  Information staging, cataloging, and discovery