{"id":67000,"date":"2022-05-16T13:54:53","date_gmt":"2022-05-16T12:54:53","guid":{"rendered":"https:\/\/www.artefact.com\/?post_type=blog&p=67000"},"modified":"2024-09-20T17:45:49","modified_gmt":"2024-09-20T16:45:49","slug":"what-does-the-future-of-data-engineering-look-like","status":"publish","type":"blog","link":"https:\/\/www.artefact.com\/zh\/blog\/what-does-the-future-of-data-engineering-look-like\/","title":{"rendered":"data \u5de5\u7a0b\u6280\u672f\u7684\u672a\u6765\u662f\u600e\u6837\u7684\uff1f"},"content":{"rendered":"

\u4f5c\u8005<\/h2><\/div>\"Image\"

\u8d1d\u52aa\u74e6-\u53e4\u620e<\/h3><\/div>

Data Engineer at Artefact<\/p>\n<\/div><\/div><\/div><\/div><\/div>

<\/span><\/a><\/span>

\u9605\u8bfb\u6211\u4eec\u7684\u6587\u7ae0<\/u><\/p>\n<\/div>

<\/span><\/div>

.<\/p>\n<\/div><\/div><\/div><\/div><\/div>

data \u5de5\u7a0b\u9886\u57df\u53d1\u5c55\u8fc5\u901f\u3002\u672c\u6587\u4ecb\u7ecd\u4e86\u6211\u8ba4\u4e3a\u5728\u672a\u6765\u51e0\u5e74\u4e2d\u5c06\u4f1a\u51f8\u663e\u7684\u4e09\u5927\u8d8b\u52bf\u3002.<\/p>\n<\/div><\/div><\/div><\/div><\/div>

\u5341\u5e74\u524d\uff0cdata \u5de5\u7a0b\u5e08\u7684\u89d2\u8272\u51e0\u4e4e\u4e0d\u5b58\u5728\u3002\u4f46\u662f\uff0c\u5bf9\u8fd9\u79cd\u7279\u6b8a\u8f6f\u4ef6\u5de5\u7a0b\u5e08\u7684\u9700\u6c42\u4e0e\u65e5\u4ff1\u589e\u3002\u968f\u7740\u8fd9\u4e00\u9886\u57df\u65e5\u8d8b\u6210\u719f\uff0c\u5de5\u7a0b\u5e08\u7684\u89d2\u8272\u4e5f\u5728\u4e0d\u65ad\u6f14\u53d8\u3002.<\/p>\n

data \u5de5\u7a0b\u5e08\u7684\u804c\u8d23\u56e0\u516c\u53f8\u800c\u5f02\uff0c\u5176\u89d2\u8272\u5728\u5404\u5730\u7684\u53d1\u5c55\u901f\u5ea6\u4e5f\u4e0d\u5c3d\u76f8\u540c\u3002\u4f46\u6211\u8ba4\u4e3a\u89d2\u8272\u7684\u53d8\u5316\u4f53\u73b0\u5728\u4e09\u4e2a\u65b9\u9762\uff1a<\/p>\n<\/div>

  • <\/i><\/span>
    Data \u5de5\u7a0b\u5e08\u5c06\u5927\u91cf\u5229\u7528 cloud \u6280\u672f\u548c SaaS \u4ea7\u54c1<\/div><\/li>
  • <\/i><\/span>
    Data \u5de5\u7a0b\u5e08\u5c06\u51cf\u5c11\u7f16\u7801\u65f6\u95f4\uff0c\u5c06\u66f4\u591a\u65f6\u95f4\u7528\u4e8e\u76d1\u63a7<\/div><\/li>
  • <\/i><\/span>
    Data \u5de5\u7a0b\u5e08\u5c06\u4ece\u529f\u80fd\u56e2\u961f\u8f6c\u4e3a\u57fa\u7840\u56e2\u961f<\/div><\/li><\/ul>

    \u8ba9\u6211\u4eec\u6765\u8be6\u7ec6\u4e86\u89e3\u4e00\u4e0b\u3002.<\/p>\n<\/div>

    Data \u5de5\u7a0b\u5e08\u5c06\u5927\u91cf\u5229\u7528 cloud \u6280\u672f\u548c SaaS \u4ea7\u54c1<\/h2><\/div>

    \u5341\u5e74\u524d\uff0c\u4f01\u4e1a\u8fd8\u4f9d\u8d56\u5185\u90e8\u57fa\u7840\u8bbe\u65bd\u6765\u5b58\u50a8 data\u3002\u8fd9\u5c31\u662f\u4e3a\u4ec0\u4e48\u00a0\u7b2c\u4e00\u6279\u5927\u578b data \u6280\u672f\u662f\u4e3a\u5185\u90e8\u90e8\u7f72\u73af\u5883\u6784\u5efa\u7684<\/strong>. .\u5728\u8fd9\u4e2a\u65f6\u4ee3\uff0cdata \u5de5\u7a0b\u5e08\u82b1\u8d39\u4e86\u5927\u91cf\u65f6\u95f4\u8c03\u6574\u673a\u5668\u914d\u7f6e\uff0c\u5374\u727a\u7272\u4e86\u521b\u9020\u4e1a\u52a1\u4ef7\u503c\u7684\u673a\u4f1a\u3002.<\/p>\n

    \u90a3\u4e48,\u00a0cloud \u63d0\u4f9b\u5546\u627f\u8bfa\u4e3a\u60a8\u63d0\u4f9b\u4ed6\u4eec\u7ba1\u7406\u7684\u670d\u52a1<\/strong>. .\u56e0\u6b64\uff0c\u60a8\u53ef\u4ee5\u4e13\u6ce8\u4e8e\u60a8\u7684\u4e1a\u52a1\u9700\u6c42\u3002\u8fd9\u5df2\u7ecf\u6539\u53d8\u4e86\u6e38\u620f\u89c4\u5219\u3002.<\/p>\n

    \u73b0\u5728\uff0c\u50cf Snowflake \u548c Databricks \u8fd9\u6837\u7684 cloud \u63d0\u4f9b\u5546\u548c\u6280\u672f\u516c\u53f8\u5df2\u7ecf\u8ba9\u5927\u578b data \u53d8\u5f97\u8f7b\u800c\u6613\u4e3e\u3002data \u751f\u6001\u7cfb\u7edf\u4e5f\u53d8\u5f97\u66f4\u52a0\u6210\u719f\u3002\u5728 data \u8d28\u91cf\u3001data governance \u6216 data \u6444\u53d6\u7b49\u7279\u5b9a\u9886\u57df\u51fa\u73b0\u4e86\u65b0\u7684 data \u521d\u521b\u516c\u53f8\u3002\u8fd9\u4e9b\u4ea7\u54c1\u4e4b\u95f4\u5b9e\u73b0\u4e86\u65e0\u7f1d\u96c6\u6210\u3002.<\/p>\n<\/div>

    <\/span><\/div>

    data \u5de5\u7a0b\u5e08\u53ea\u9700\u4f7f\u7528 Apache \u57fa\u91d1\u4f1a\u63d0\u4f9b\u7684\u4e00\u79cd\u5de5\u5177\u5c31\u80fd\u6ee1\u8db3\u7279\u5b9a\u9700\u6c42\u7684\u65f6\u4ee3\u5df2\u7ecf\u4e00\u53bb\u4e0d\u590d\u8fd4\u4e86\u3002\u4ed6\u4eec\u6709\u65e0\u6570\u7684\u5de5\u5177\u6765\u505a\u540c\u6837\u7684\u4e8b\u60c5\u3002\u5982\u4eca\uff0cdata \u5de5\u7a0b\u5e08\u6709\u8d23\u4efb\u9009\u62e9\u6700\u597d\u7684\u5de5\u5177\u3002\u56e0\u6b64\uff0c\u4ed6\u4eec\u9700\u8981\u5bf9\u751f\u6001\u7cfb\u7edf\u6709\u5145\u5206\u7684\u4e86\u89e3\uff0c\u77e5\u9053\u5982\u4f55\u8fdb\u884c\u57fa\u51c6\u6d4b\u8bd5\u548c\u9009\u62e9\u76f8\u5173\u7684\u51b3\u7b56\u6807\u51c6\u3002.<\/p>\n

    \u4e3a\u6b63\u786e\u7684\u5de5\u4f5c\u9009\u62e9\u6b63\u786e\u7684\u5de5\u5177\u5e76\u975e\u6613\u4e8b\u3002\u4f46\u6574\u5408\u5de5\u5177\u4ee5\u5f62\u6210\u4e00\u81f4\u7684 data platform \u4e5f\u662f\u4e00\u9879\u6311\u6218\u3002\u4e00\u4e9b data \u5de5\u7a0b\u5e08\u5df2\u7ecf\u5229\u7528\u57fa\u7840\u67b6\u6784\u5373\u4ee3\u7801\u6765\u7ec4\u88c5\u8fd9\u4e9b\u7816\u5757\u5e76\u81ea\u52a8\u90e8\u7f72\u57fa\u7840\u67b6\u6784\u3002\u6211\u8ba4\u4e3a\u8fd9\u5c06\u6210\u4e3a\u4e00\u9879\u5fc5\u5907\u6280\u80fd\u3002.<\/p>\n<\/div>

    Data \u5de5\u7a0b\u5e08\u5c06\u51cf\u5c11\u7f16\u7801\u65f6\u95f4\uff0c\u5c06\u66f4\u591a\u65f6\u95f4\u7528\u4e8e\u76d1\u63a7<\/h2><\/div>

    data \u5de5\u7a0b\u5e08\u4f7f\u7528 Scala \u548c Spark \u5f00\u53d1\u590d\u6742 ETL \u7ba1\u9053\u7684\u65f6\u4ee3\u4f3c\u4e4e\u5df2\u7ecf\u8fc7\u53bb\u3002.<\/p>\n

    \u5bf9\u4e8e\u63d0\u53d6\u90e8\u5206\uff0c\u4f60\u73b0\u5728\u53ef\u4ee5\u4f7f\u7528 Airbyte \u7b49\u6280\u672f\u6765\u5b89\u6392\u4ece\u8bb8\u591a\u4e0d\u540c\u6765\u6e90\u63d0\u53d6\u7684\u5de5\u4f5c\u3002\u52a0\u8f7d\u90e8\u5206\u4e0d\u518d\u662f\u75db\u70b9\u3002\u4f8b\u5982\uff0cSnowflake \u53ef\u4ee5\u901a\u8fc7\u4e00\u6761 SQL \u547d\u4ee4\u8f7b\u677e\u5730\u5c06\u6587\u4ef6\u4ece blob \u5b58\u50a8\u52a0\u8f7d\u5230\u8868\u4e2d\u3002.
    \n\u5c31\u8f6c\u53d8\u6b65\u9aa4\u800c\u8a00\uff0cdbt \u5e26\u6765\u4e86\u4e00\u79cd\u65b0\u7684\u6a21\u5f0f\uff0c\u5728\u8fd9\u79cd\u6a21\u5f0f\u4e2d\uff0c\u4f60\u53ef\u4ee5\u00a0\u5728\u60a8\u7684 data \u4ed3\u5e93\u4e2d\u6539\u9020\u60a8\u7684 data<\/strong>\u00a0\u4f7f\u7528 SQL \u4f5c\u4e3a\u4e3b\u8981\u8bed\u8a00\u3002\u8be5\u7cfb\u7edf\u00a0\u5b8c\u6210\u4ece ETL \u5230 ELT \u7684\u8f6c\u53d8<\/strong>.<\/p>\n<\/div>

    \u56e0\u6b64\uff0c\u90e8\u7f72\u5de5\u4f5c\u6d41\u7a0b\u4ece\u672a\u5982\u6b64\u7b80\u5355\uff0c\u6211\u4eec\u53ef\u4ee5\u8bf4\u00a0\u8c22\u8c22<\/em>\u00a0\u5230\u73b0\u4ee3 data \u534f\u8bae\u6808\u3002\u73b0\u4ee3 data \u534f\u8bae\u6808\u662f\u4e00\u5957\u6280\u672f\uff0c\u65e8\u5728\u964d\u4f4e data \u5de5\u4f5c\u6d41\u7a0b\u7684\u590d\u6742\u6027\uff0c\u63d0\u9ad8 data \u56e2\u961f\u7684\u5de5\u4f5c\u901f\u5ea6\u3002\u6709\u4e86\u73b0\u4ee3 data \u534f\u8bae\u6808\uff0cdata \u5206\u6790\u5458\u73b0\u5728\u53ef\u4ee5\u81ea\u4e3b\u5de5\u4f5c\u3002\u4ed6\u4eec\u4e0d\u518d\u9700\u8981 data \u5de5\u7a0b\u5e08\u7684\u5e2e\u52a9\u6765\u6536\u96c6\u548c\u8f6c\u6362\u539f\u59cb data\u3002\u4f46\u8fd9\u662f\u5426\u610f\u5473\u7740 data \u56e2\u961f\u4e0d\u518d\u9700\u8981 data \u5de5\u7a0b\u5e08\uff1f\ud83d\ude1f
    \n\u6211\u53ef\u80fd\u6709\u504f\u89c1\uff0c\u4f46\u6211\u8ba4\u4e3a\u7b54\u6848\u662f\u5426\u5b9a\u7684\u3002.<\/p>\n<\/div>

    \"My<\/span><\/div>

    \u6211\u7684\u731c\u6d4b\u662f\u00a0data \u5de5\u7a0b\u5e08\u7684\u89d2\u8272\u5c06\u671d\u7740\u66f4\u52a0\u9762\u5411\u884c\u52a8\u7684\u65b9\u5411\u53d1\u5c55<\/strong>. .\u4e0b\u4e00\u4ee3 data \u5de5\u7a0b\u5e08\u5c06\u81f4\u529b\u4e8e\u63d0\u9ad8\u6574\u4e2a\u516c\u53f8\u7684 data \u53ef\u9760\u6027\u3002\u4ed6\u4eec\u7684\u804c\u8d23\u662f<\/p>\n<\/div>