neo4j节点添加性能测试郝伟 2021/02/04 [TOC]

1. 测试环境

CPU：Intel(R) Xeon(R) CPU E5-2630 @ 2.30GHz （2012Q1发布） MEM：MemTotal: 16,266,540 kB，MemFree: 356,032 kB，MemAvailable: 7,566,740 kB

2. 多节点同时插入测试，插入性能可以提高约12倍，插入速度达到500个节点/秒

通过调研，参考此网站发现可以通过 Create (n1), (n2),... ，发现可以实现多节点的同时插入。注：由于CPU性能比较弱，所以速度应该还有一定的提升空间。

2.1. 结果

以上代码是每批添加10个节点的，结果如下：

node count: 1000, time eclipsed:  4.3s, adding speed: 230 nodes per sec.
node count: 10000, time eclipsed:  45.6s, adding speed: 219 nodes per sec.

如果每批改成100个，结果如下：

node count: 10000, time eclipsed:  20.8s, adding speed: 481 nodes per sec.
node count: 100000, time eclipsed:  182.4s, adding speed: 548 nodes per sec.

2021/02/06 补充：在AMD 4700U @ 2.00 GHz + 16GB 红米笔记本测试性能达到1500节点每秒：

node count:  10000, time eclipsed:  6.5s, adding speed: 1548 nodes per sec.
node count: 100000, time eclipsed:  67.7s, adding speed: 1478 nodes per sec.

2.2. 结论

通过修改每次添加节点的数量，可以实现每秒约500个节点的添加，相当于每天4000万条。对于目标数据的约240万条数据（43万个节点和196万条关系）插入时间约为 1.44 小时。 PS. 后面又调整count和batch_count进行了些测试，发现速度基本稳定在500条/s左右

2.3. 测试代码


  public static void AddMutipleNodesTest(int count = 1_000, int batch_count=10)
  {
      // 建立连接与会话
      var driver = GraphDatabase.Driver("bolt://192.168.3.178:7687", AuthTokens.Basic("neo4j", "123456"));
      var session = driver.Session();

      DateTime startTime = DateTime.Now;
      for (int i = 0; i < count; i++)
      {
          // 定义长度为batch_count的节点的 CQL 插入语句
          string[] nodes = new string[batch_count];
          for (int j = 0; j < nodes.Length; j++)
              nodes[j] = "(n$id:TestNode:Greeting{name: 'T & G _$id'})".Replace("$id", $"{i}_{j}");
          // 创建完整的 cql 插入语句并执行
          string cql = ("CREATE " + string.Join(",", nodes)).Replace("'", "\"");
          session.Run(cql);
      }
      DateTime endTime = DateTime.Now;
      double time = (endTime - startTime).TotalSeconds;
      double speed = count * batch_count / time;
      Console.WriteLine("node count: {0}, time eclipsed:  {1:0.0}s, adding speed: {2:0} nodes per sec.", count * batch_count, time, speed);

      // 释放资源
      session.Dispose();
      driver.CloseAsync();
  }

3. 节点插入和删除测试结果，插入节点速度为40个节点/秒

3.1. 输出结果

# 1K和10K个节点速度测试结果
node count: 1000 , time eclipsed:  24.5s, adding speed: 40.9 nodes per sec.
node count: 10000, time eclipsed: 268.4s, adding speed: 37.3 nodes per sec. 

# 删除节点却很快，使用命令 match (n:TestNode) delete n 仅用时0.662秒
Deleted 14110 nodes, completed after 662 ms.

3.2. 结论

通过远程单个节点添加的方式，大约每秒可以添加40个节点，而删除的每秒可以删除2万个节点。

3.3. 测试代码

using Neo4j.Driver;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace New4j_HelloWorld
{
    class Program
    {
  static void Main(string[] args)
  {
      AddNodesTest();
      //Student s1 = new Student() { id = "100001", name = "Jack", sex = "M" };
      //Student s2 = new Student() { id = "100002", name = "Lucy", sex = "F" };
      //Relation r1 = new Relation() { id = "100003", name = "isFriendTo", node1 = s1, node2 = s2 };
      //Console.WriteLine(s2.To(s1));
      //Console.WriteLine(s1.To(s2));

      // Console.WriteLine(r1);

      // node count: 1000 , time eclipsed:  24.5s, adding speed: 40.9 nodes per sec.
      // node count: 10000, time eclipsed: 268.4s, adding speed: 37.3 nodes per sec. 
      // 删除节点却很快，使用命令 match (n:TestNode) delete n 仅用时0.662秒
      // Deleted 14110 nodes, completed after 662 ms.
  }

  public static void AddNodesTest(int count = 1_000)
  {
      var driver = GraphDatabase.Driver("bolt://192.168.3.178:7687", AuthTokens.Basic("neo4j", "123456"));
      var session = driver.Session();

      DateTime dt1 = DateTime.Now;
      for (int i = 0; i < count; i++)
      {
   // create(student:Student{id:1, name: "李雷"})
   session.Run("CREATE (a:TestNode{name: \"$name\"})".Replace("$name", "node" + i));
      }
      DateTime dt2 = DateTime.Now;
      double time = (dt2 - dt1).TotalSeconds;
      double speed = count / time;
      Console.WriteLine("node count: {0} , time eclipsed:  {1:0.0}s, adding speed: {2:0.0} nodes per sec.", count, time, speed);

      session.Dispose();
      driver.CloseAsync();
  }
    }
}

// 插入节点语法
// CREATE (n9_0:TestNode:Greeting{name: "T & G _9_0"}),(n9_1:TestNode:Greeting{name: "T & G _9_1"}),(n9_2:TestNode:Greeting{name: "T & G _9_2"}),(n9_3:TestNode:Greeting{name: "T & G _9_3"}),(n9_4:TestNode:Greeting{name: "T & G _9_4"}),(n9_5:TestNode:Greeting{name: "T & G _9_5"}),(n9_6:TestNode:Greeting{name: "T & G _9_6"}),(n9_7:TestNode:Greeting{name: "T & G _9_7"}),(n9_8:TestNode:Greeting{name: "T & G _9_8"}),(n9_9:TestNode:Greeting{name: "T & G _9_9"}),(n9_10:TestNode:Greeting{name: "T & G _9_10"}),(n9_11:TestNode:Greeting{name: "T & G _9_11"}),(n9_12:TestNode:Greeting{name: "T & G _9_12"}),(n9_13:TestNode:Greeting{name: "T & G _9_13"}),(n9_14:TestNode:Greeting{name: "T & G _9_14"}),(n9_15:TestNode:Greeting{name: "T & G _9_15"}),(n9_16:TestNode:Greeting{name: "T & G _9_16"}),(n9_17:TestNode:Greeting{name: "T & G _9_17"}),(n9_18:TestNode:Greeting{name: "T & G _9_18"}),(n9_19:TestNode:Greeting{name: "T & G _9_19"}),(n9_20:TestNode:Greeting{name: "T & G _9_20"}),(n9_21:TestNode:Greeting{name: "T & G _9_21"}),(n9_22:TestNode:Greeting{name: "T & G _9_22"}),(n9_23:TestNode:Greeting{name: "T & G _9_23"}),(n9_24:TestNode:Greeting{name: "T & G _9_24"}),(n9_25:TestNode:Greeting{name: "T & G _9_25"}),(n9_26:TestNode:Greeting{name: "T & G _9_26"}),(n9_27:TestNode:Greeting{name: "T & G _9_27"}),(n9_28:TestNode:Greeting{name: "T & G _9_28"}),(n9_29:TestNode:Greeting{name: "T & G _9_29"}),(n9_30:TestNode:Greeting{name: "T & G _9_30"}),(n9_31:TestNode:Greeting{name: "T & G _9_31"}),(n9_32:TestNode:Greeting{name: "T & G _9_32"}),(n9_33:TestNode:Greeting{name: "T & G _9_33"}),(n9_34:TestNode:Greeting{name: "T & G _9_34"}),(n9_35:TestNode:Greeting{name: "T & G _9_35"}),(n9_36:TestNode:Greeting{name: "T & G _9_36"}),(n9_37:TestNode:Greeting{name: "T & G _9_37"}),(n9_38:TestNode:Greeting{name: "T & G _9_38"}),(n9_39:TestNode:Greeting{name: "T & G _9_39"}),(n9_40:TestNode:Greeting{name: "T & G _9_40"}),(n9_41:TestNode:Greeting{name: "T & G _9_41"}),(n9_42:TestNode:Greeting{name: "T & G _9_42"}),(n9_43:TestNode:Greeting{name: "T & G _9_43"}),(n9_44:TestNode:Greeting{name: "T & G _9_44"}),(n9_45:TestNode:Greeting{name: "T & G _9_45"}),(n9_46:TestNode:Greeting{name: "T & G _9_46"}),(n9_47:TestNode:Greeting{name: "T & G _9_47"}),(n9_48:TestNode:Greeting{name: "T & G _9_48"}),(n9_49:TestNode:Greeting{name: "T & G _9_49"}),(n9_50:TestNode:Greeting{name: "T & G _9_50"}),(n9_51:TestNode:Greeting{name: "T & G _9_51"}),(n9_52:TestNode:Greeting{name: "T & G _9_52"}),(n9_53:TestNode:Greeting{name: "T & G _9_53"}),(n9_54:TestNode:Greeting{name: "T & G _9_54"}),(n9_55:TestNode:Greeting{name: "T & G _9_55"}),(n9_56:TestNode:Greeting{name: "T & G _9_56"}),(n9_57:TestNode:Greeting{name: "T & G _9_57"}),(n9_58:TestNode:Greeting{name: "T & G _9_58"}),(n9_59:TestNode:Greeting{name: "T & G _9_59"}),(n9_60:TestNode:Greeting{name: "T & G _9_60"}),(n9_61:TestNode:Greeting{name: "T & G _9_61"}),(n9_62:TestNode:Greeting{name: "T & G _9_62"}),(n9_63:TestNode:Greeting{name: "T & G _9_63"}),(n9_64:TestNode:Greeting{name: "T & G _9_64"}),(n9_65:TestNode:Greeting{name: "T & G _9_65"}),(n9_66:TestNode:Greeting{name: "T & G _9_66"}),(n9_67:TestNode:Greeting{name: "T & G _9_67"}),(n9_68:TestNode:Greeting{name: "T & G _9_68"}),(n9_69:TestNode:Greeting{name: "T & G _9_69"}),(n9_70:TestNode:Greeting{name: "T & G _9_70"}),(n9_71:TestNode:Greeting{name: "T & G _9_71"}),(n9_72:TestNode:Greeting{name: "T & G _9_72"}),(n9_73:TestNode:Greeting{name: "T & G _9_73"}),(n9_74:TestNode:Greeting{name: "T & G _9_74"}),(n9_75:TestNode:Greeting{name: "T & G _9_75"}),(n9_76:TestNode:Greeting{name: "T & G _9_76"}),(n9_77:TestNode:Greeting{name: "T & G _9_77"}),(n9_78:TestNode:Greeting{name: "T & G _9_78"}),(n9_79:TestNode:Greeting{name: "T & G _9_79"}),(n9_80:TestNode:Greeting{name: "T & G _9_80"}),(n9_81:TestNode:Greeting{name: "T & G _9_81"}),(n9_82:TestNode:Greeting{name: "T & G _9_82"}),(n9_83:TestNode:Greeting{name: "T & G _9_83"}),(n9_84:TestNode:Greeting{name: "T & G _9_84"}),(n9_85:TestNode:Greeting{name: "T & G _9_85"}),(n9_86:TestNode:Greeting{name: "T & G _9_86"}),(n9_87:TestNode:Greeting{name: "T & G _9_87"}),(n9_88:TestNode:Greeting{name: "T & G _9_88"}),(n9_89:TestNode:Greeting{name: "T & G _9_89"}),(n9_90:TestNode:Greeting{name: "T & G _9_90"}),(n9_91:TestNode:Greeting{name: "T & G _9_91"}),(n9_92:TestNode:Greeting{name: "T & G _9_92"}),(n9_93:TestNode:Greeting{name: "T & G _9_93"}),(n9_94:TestNode:Greeting{name: "T & G _9_94"}),(n9_95:TestNode:Greeting{name: "T & G _9_95"}),(n9_96:TestNode:Greeting{name: "T & G _9_96"}),(n9_97:TestNode:Greeting{name: "T & G _9_97"}),(n9_98:TestNode:Greeting{name: "T & G _9_98"}),(n9_99:TestNode:Greeting{name: "T & G _9_99"})

4. 扩展

根据此网址提供的内容，共有4种方式导入节点，其中最慢的是 Create。

Load CSV 指令导入 LOAD CSV FROM "file:///actors.csv" AS line CREATE (a:actors{personId:line[0],name:line[1],type:line[2]}) 将文件 actors.csv 放入neo4j设备目录下的import文件夹下。

neo4j节点添加性能测试 2021/02/04